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Foreword 


The  Agency  for  Health  Care  Policy  and  Research  (AHCPR),  through 
its  Office  of  Science  and  Data  Development,  has  been  working  to  provide 
greater  access  to  health  care  data  bases,  uniform  definitions,  common 
reporting  formats  and  linkages,  and  standards  to  ensure  security,  confiden- 
tiality, accuracy,  and  appropriate  maintenance  of  data. 

This  literature  review  should  provide  medical  effectiveness  researchers 
with  a  valuable  tool  for  identifying  and  accessing  automated  medical 
records  on  ambulatory  care. 

J.  Jarrett  Clinton,  M.D.,  M.P.H. 
Administrator 

Agency  for  Health  Care  Policy  and  Research 
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Preface 


Medical  effectiveness  studies  that  involve  use  of  automated  ambulatory 
medical  records  constitute  an  emerging  field.  To  date,  there  has  been  little 
guidance  to  assist  researchers  in  locating  previous  studies  that  employed  the 
type  of  data  of  interest.  This  literature  review  was  designed  to  help  fill  that 
void.  It  represents  a  systematic  and  objective  description  of  the  available  lit- 
erature on  medical  treatment  effectiveness  in  ambulatory  settings  in  which 
the  researchers  have  relied  substantially  on  computerized  medical  records. 
The  review  includes  studies  that  involved  the  use  of  computerized  records 
to  identify  practice  patterns  (particularly  the  effects  of  these  patterns  on 
patient  survival,  quality  of  life,  and  costs  of  care). 

In  collecting  and  describing  these  resources,  emphasis  was  placed  on  the 
characteristics  of  the  record  system  and  the  methodology  of  record  search 
and  data  extraction.  Based  on  information  derived  from  the  literature 
review,  the  reader  will  be  able  to  identify  computerized  ambulatory  record 
systems  suitable  for  research,  including  systems  in  use  in  countries  other 
than  the  United  States. 

This  literature  review  provides  a  basis  for  discussions  of  linkages 
between  "certain  types  of  data"  to  proceed  based  on  the  general  content 
and  levels  of  linkages  possible  with  existing  data  bases.  The  descriptions  of 
the  data  bases  also  will  facilitate  an  analysis  of  the  usefulness  of  combining 
the  information  contained  within  different  data  bases  to  address  additional 
medical  effectiveness  issues. 


Harvey  A. 

Schwartz,  Ph.D. 
Project  Officer 
Office  of  Science  and 

Data  Development 
Agency  for  Health  Care 

Policy  and  Research 


J.  Michael  Fitzmaurice, 
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Director 
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Executive  Summary 


A  comprehensive  review  of  the  literature  was  per- 
formed to  identify  automated  ambulatory  medical 
records  systems  (AAMRSs),  issues  relating  to 
AAMRSs,  and  studies  of  medical  effectiveness  that 
have  used  automated  medical  records.  In  addition 
to  system  descriptions,  two  major  categories  of  liter- 
ature and  materials  were  identified:  (1)  empirical 
studies  that  are  based  primarily  on  AAMRSs  and 
(2)  descriptive  literature  on  policy,  methods,  and 
technology,  using  MEDLINE,  SOCIOFILE,  CAT- 
LINE,  the  Directory  of  Published  Proceedings  and 
Proceedings  in  Print,  as  well  as  Congressional 
Record  Abstracts  (CREC),  PAIS,  and  LEXIS. 
Uncataloged  materials — such  as  recent  journals, 
books,  abstracts,  and  conference  proceedings — also 
were  reviewed,  and  personal  contacts  were  explored 
to  identify: 

•  AAMRS  descriptions. 

•  Studies  using  computerized  medical  records  in 
ambulatory  care  settings,  especially  those  detail- 
ing practice  patterns,  process  of  care,  outcome, 
and  costs,  in  addition  to  those  using  computer- 
ized medical  records  that  can  be  adapted  to 
ambulatory  care  settings. 

•  Outcome  studies  including  survival,  life  quality, 
satisfaction,  and  health  status. 


•  Technology  issues  including  hardware,  software, 
research  strategies,  retrieval  mechanisms,  and 
query  languages. 

•  Methodological  issues  including  research  ques- 
tions, definitions,  operations  manuals,  quality 
control  studies,  analytic  techniques,  and  statisti- 
cal considerations. 

Materials  were  gathered  and  coded  and  are 
described  in  this  review.  Although  medical  effec- 
tiveness studies  that  use  automated  ambulatory 
medical  records  are  of  major  interest  to  the  Agency 
for  Health  Care  Policy  and  Research  (AHCPR), 
only  a  few  such  studies  were  found,  perhaps  because 
the  field  is  an  emerging  one. 

Chapter  1  of  this  literature  review  covers  ambu- 
latory medical  record  systems,  and  Chapter  2 
reviews  medical  effectiveness  and  health  services 
research  studies  and  issues,  as  well  as  quality  assur- 
ance articles.  Chapter  3  describes  the  use  of  auto- 
mated medical  records  for  decisionmaking.  Chapter 
4  addresses  legal,  confidentiality,  and  privacy  issues; 
and  Chapter  5  identifies  methodological  issues  asso- 
ciated with  the  use  of  automated  ambulatory  med- 
ical records.  Chapter  6  describes  selected  automated 
ambulatory  medical  record  systems. 
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1  ■  Ambulatory  Care  Records  Systems  and  Data  Bases 


Introduction 

Ambulatory  care  services  are  provided  to  non- 
hospitalized  patients  in: 

•  Hospital-based  clinics  and  programs. 

•  Homes,  through  home  health  agencies. 

•  Private  physicians1  offices. 

•  Prehospital  emergency  medical  systems. 

•  Community  health  clinics. 

•  Free-standing  clinics. 

•  Health  maintenance  organizations  (HMOs). 

Because  of  the  numerous  changes  in  the  health  care 
system,  the  most  important  of  which  is  the  introduc- 
tion of  the  Federal  prospective  payment  movement, 
there  has  been  a  shift  of  care  from  the  hospital  inpa- 
tient setting  to  the  ambulatory  care  setting.  Evi- 
dence of  the  shift  is  shown  by  the  following  (Matson 
and  McDougall,  1990): 

•  The  number  of  visits  to  outpatient  sites  exceeds 
the  number  of  short-term,  inpatient  care  days  on 
a  nationwide  basis. 

•  Total  hospital  revenues  from  outpatient  services 
increased  from  12  percent  in  1983  to  26  percent 
in  1988,  an  increase  of  117  percent. 

•  It  is  predicted  that  by  1995  the  majority  of  hospi- 
tals will  derive  50  percent  of  their  revenue  from 
outpatient  services. 

•  The  number  of  free-standing  ambulatory  care 
providers  is  expected  to  increase  from  19,516  in 
1980  to  39,550  in  1990. 

It  is  anticipated  that  the  increased  use  of  ambu- 
latory care  services  will  dictate  the  need  for  infor- 
mation systems  with  integration  among  ambulatory, 
inhospital,  and  ancillary  hospital  systems,  and  this 
area  should  be  one  of  growth  in  the  1990s.  The  ben- 
efits of  computers  to  health  care  delivery  have  been 
described,  and  the  increased  use  of  computers  and 
information  systems  in  the  health  care  sector  gener- 
ally, and  the  medical  record  specifically,  has  been 


forecasted  since  the  late  1960s  (Shortliffe  and  Per- 
reault,  1990). 

The  medical  record  summarizes  a  patient's  med- 
ical history  and  documents  the  observations,  diag- 
nostic conclusions,  and  plans  of  health  personnel 
that  is  supposed  to  support  patient  care  and  clinical 
research  (Shortliffe  and  Barnett,  1990).  Several 
arguments  support  the  need  for  automated  ambula- 
tory care  medical  records.  Although  the  contents  of 
the  hospital  medical  record  are  mandated,  the  con- 
tents of  the  ambulatory  record  may  not  be  as  strin- 
gently regulated  (Mesel  and  Wirtschafter,  1976). 
Several  authors  note  that  paper  ambulatory  care 
records  frequently  contain  poorly  organized  data, 
inaccurate  information  including  diagnostic  coding, 
and  lack  of  documentation  on  key  aspects  of  care. 
Acknowledged  weaknesses  include: 

•  Lack  of  standardization  in  content  and  format. 

•  Inaccessibility. 

•  Incompleteness. 

•  Inaccuracies. 

In  reaction  to  the  difficulties  with  the  paper 
medical  record,  Weed  introduced  the  Problem-Ori- 
ented Medical  Record  (POMR)  in  1968  that  struc- 
tured the  medical  record  by  problem  rather  than  by 
strict  chronology.  In  the  early  1970s,  Fries  intro- 
duced the  Time-Oriented  Record  (TOR)  that  col- 
lects patient  encounters  in  an  array  over  time,  by 
date,  so  problems  can  be  viewed  across  time  (Fries, 
1972).  These  approaches  to  the  medical  record 
became  the  basis  of  early  automated  medical 
records  systems.  For  example,  the  TOR  became  the 
basis  of  other  record  systems,  including  the  Ameri- 
can Rheumatism  Association  Medical  Information 
System  (ARAMIS),  discussed  below. 

Using  the  POMR,  Weed  and  Zimny  (1989) 
developed  PROMIS  (the  Problem-Oriented  Med- 
ical Information  System).  This  system — which  was 
to  replace  all  paper  recordkeeping,  was  technologi- 
cally advanced,  and  included  a  touch  screen  and 
other  innovative  features — is  no  longer  used.  How- 
ever, the  concept  of  the  problem-oriented  record  is 
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considered  to  be  a  major  contribution  by  Weed. 
More  recently,  Weed  (1991)  developed  a  problem- 
knowledge  coupler  for  microcomputers  to  assist 
clinicians  in  organizing  patient  data  in  a  variety  of 
care  settings.  This  system  permits  the  clinician  to 
build  a  computer-based  problem-oriented  patient 
record  and  to  couple  that  record  with  knowledge 
derived  from  the  literature. 

Many  authors  have  argued  that  an  automated 
ambulatory  medical  record  would  improve  care  and 
provide  diagnostic  support  (Jenkin,  Cheezum, 
Essick,  and  others,  1978).  Further,  an  automated 
record  would  improve  productivity,  accessibility, 
and  accuracy.  More  specifically,  the  automated 
ambulatory  medical  record  improves:  (1)  organiza- 
tion and  reporting  of  patient  information;  (2)  effi- 
ciency and  accuracy  of  clinical  decisionmaking;  (3) 
guidance  for  future  policies  and  practice;  and  (4) 
data  retrieval  and  organization  (McDonald  and 
Tierney,  1988).  The  Institute  of  Medicine  (IOM)  is 
supporting  the  development  of  a  computer-based 
medical  record  and  has  recently  completed  a  study 
documenting  its  necessity  (Dick  and  Steen,  1991 ). 

In  addition  to  practice  benefits,  AAMRSs  can 
support  health  services  and  clinical  research.  They 
can  be  used  for  epidemiologic  studies,  risk  analyses, 
drug  surveillance,  resource  consumption  and  quality 
assurance  analyses,  and  studies  of  variation  in  clini- 
cal delivery  (Tierney  and  McDonald,  1991).  With 
regard  to  the  latter  category,  Roper,  Winkenwerder, 
Hackbarth,  and  Krakauer  (1988)  and  Ellwood 
(1988)  recognized  the  need  for  systematically 
derived  data  with  which  to  evaluate  the  effective- 
ness of  medical  care  (Agency  for  Health  Care  Pol- 
icy and  Research  [AHCPR],  1991). 

Although  the  importance  of  automated  medical 
records  has  been  recognized  and  discussed  for  25 
years,  automated  ambulatory  records  systems  are 
used  in  relatively  few  institutions  (Shortliffe  and 
Barnett,  1990).  Automation  in  outpatient  settings 
has  focused  on  applications  that  are  often  tied  to 
revenue  and  administrative  tasks  such  as  billing, 
claims  submissions,  scheduling,  and  registration. 
These  applications  are  also  less  expensive  than  a 
medical  records  system.  As  of  1989,  71,000  vendors 
offered  commercial  ambulatory  care  information 
systems  (Matson  and  McDougall,  1990). 

Some  authors  (Mayo  and  Ball,  1988)  identified  an 
extensive  list  of  features  in  office  practice  systems: 


•  Automated  patient  recall  for  followup. 

•  Medical  records. 

•  Prepaid  plan  accounting  and  referral  tracking. 

•  Medical/surgical  knowledge  bases. 

•  Computer-assisted  instruction. 

•  Appointment  scheduling. 

•  Patient  care  plan  tracking. 

•  Utilization  review. 

•  Management  statistics. 

•  Claims  submissions. 

Few  vendors,  however,  offer  such  a  comprehen- 
sive range  of  products,  and  many  do  not  seem  to 
comprehend  the  need  for  AAMRSs  (Dick  and 
Steen,  1991;  Matson  and  McDougall,  1990).  The 
IOM  conducted  a  survey  of  12  vendors  associated 
with  clinical  information  systems.  Findings  indicate 
that  vendors  are  generally  pessimistic  about  physi- 
cians1 and  nurses'  interest  in  entering  data.  Other 
vendor-identified  impediments  to  the  success  of 
AAMRSs  include  the  cost  of  the  system  and  resis- 
tance to  change  in  the  health  care  industry.  The 
need  for  interdepartmental  sharing  of  information 
and  the  lack  of  standards  to  manage  such  sharing 
further  decreases  the  likelihood  of  success  (Dick 
and  Steen,  1991).  It  is  thought  that  the  widespread 
use  of  well-documented  and  supported  AAMRSs 
based  on  current  technology  will  depend  on  com- 
mercial vendors.' 

Barriers  to  Automated  Ambulatory 
Medical  Records  Systems 

Vendors  and  authors  have  pointed  out  barriers 
to  AAMRSs.  A  frequently  cited  barrier  is  difficulty 
with  the  machine-person  interface.  For  data  to  be 
accessible  for  retrieval,  they  must  be  structured 
rather  than  gathered  in  the  free-form  text  that  char- 
acterizes the  familiar  paper  record.  Until  voice- 
recognition  is  perfected,  many  physicians  will  resist 
changing  from  their  current  practices  to  filling  out 
structured  forms.  Another  problem  is  the  electronic 
collection  of  data  from  various  services  within  a 
hospital  or  within  the  health  care  system,  such  as 
laboratories.  If  the  interface  involves  two  discrete 


1  Robert  C.  Sherrick,  M.D.,  George  Washington  University, 
personal  communication,  September  1991. 
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applications  within  the  same  system,  interface  prob- 
lems are  minimal.  If  the  interface  is  between  two 
different  hardware/software  systems,  however,  it  is 
problematic  and  still  a  challenge  for  systems  engi- 
neers (Whiting-O'Keefe,  Whiting,  and  Henke, 
1988).  Other  barriers,  mentioned  in  the  past  but  not 
as  relevant  today,  are  the  costs  of  data  storage, 
which  have  decreased,  and  fear  or  discomfort  with 
computers,  which  also  has  diminished  as  computers 
have  become  part  of  everyday  life.  Masys  (1989), 
however,  recently  stated  that  the  "fully  electronic 
clinical  record  is  a  costly  and  elusive  goal  for  most 
health  care  organizations  because  of  technical, 
financial,  and  sociologic  constraints." 

Existing  Automated  Ambulatory 
Medical  Records  Systems 

Although  AAMRSs  are  not  commonplace,  they 
have  been  developed  and  implemented  in  academic 
settings  in  a  number  of  institutions  since  the  late 
1960s  and  early  1970s.  Several  of  these  systems, 
such  as  TMR  (The  Medical  Record)  and  COSTAR 
(Computer-Stored  Ambulatory  Record),  were 
implemented  either  to  replace  the  paper  medical 
record  because  of  mentioned  difficulties  in  an  out- 
patient setting  or  to  become  part  of  a  hospital  sys- 
tem. Several  of  the  following  systems  were  stimu- 
lated by  funding  from  the  National  Center  for 
Health  Services  Research  and  Health  Care  Technol- 
ogy Assessment  (NCHSR),  the  predecessor  of 
AHCPR.  These  AAMRSs  include  COSTAR,  origi- 
nally at  the  Harvard  Community  Health  Plan 
(HCHP)  (Barnett,  Justice,  Somand,  and  others, 
1979);  TMR  at  Duke  University  Medical  Center 
(Stead  and  Hammond,  1988);  HELP  (the  Health 
Evaluation  Through  Logical  Processing)  at  the  Lat- 
ter Day  Saints  Hospital  in  Salt  Lake  City;  RMRS 
(the  Regenstrief  Medical  Records  System)  at  the 
University  of  Indiana  Medical  Center  (McDonald, 
Tierney,  Martin,  and  others,  1990);  and  the  system 
at  Beth  Israel  Hospital  in  Boston  (Bleich,  Beckley, 
Horowitz,  and  others,  1985).  These  systems  were 
implemented  about  20  years  ago,  have  evolved  as 
the  technology  has  changed,  house  thousands  of 
patient  records  with  data  collected  longitudinally, 
support  retrieval  mechanisms,  and  have  been  the 
basis  for  clinical  and  outcomes  studies.  Their 
founders  have  become  leaders  in  the  medical  infor- 
matics field  and  are  insightful  regarding  the  techni- 


cal and  pragmatic  issues  relating  to  the  implementa- 
tion of  AAMRSs. 

Often  linked  with  this  group  is  STOR  (Summary 
Time-Oriented  Record)  at  the  University  of  Califor- 
nia at  San  Francisco  (UCSF)  Medical  Center  (Whit- 
ing-O'Keefe, Whiting,  and  Henke,  1988)  and 
THERESA  at  Grady  Memorial  Hospital,  the  teach- 
ing hospital  for  Emory  University  Medical  Center 
(Walker,  1989).  STOR  and  THERESA  are  more 
recent  systems  that  have  incorporated  distributed 
and/or  local  area  network  (LAN)  technology.  These 
systems  share  common  attributes  including: 

•  Data  dictionaries  that  store  the  names  and  char- 
acteristics of  their  data  elements. 

•  Data  collection  over  time  that  provides  a  longi- 
tudinal picture  of  the  patient. 

•  Data  retrieval  capabilities  to  support  patient 
care  and  research  applications. 

•  Selective  storage  of  information. 

•  Concern  with  data  security. 

With  regard  to  selective  storage  of  information, 
these  AAMRSs  do  not  contain  all  details  that 
describe  an  observation  but  only  keep  clinically  rel- 
evant information.  The  existing  systems  (described 
in  Chapter  2  of  this  review)  also  differ  in  the  follow- 
ing ways: 

•  The  hardware  and  software  on  which  they  are 
built. 

•  The  kinds  of  information  they  house. 

•  The  vocabulary  they  use  for  storing  information. 

•  The  user  interface  (who  enters  the  data). 

•  The  type  of  prompts  for  clinical  detail. 

•  Their  record  structure. 

•  Their  inpatient  or  outpatient  emphasis  (McDon- 
ald and  Tierney,  1988). 

Another  AAMRS  that  has  been  in  operation 
since  1968  is  PCIS  (Patient  Care  Information  Sys- 
tem), which  serves  Alaska  Natives  and  the  Papago 
Indian  Reservation  in  Tucson,  Arizona  (Levinson 
and  Dambro,  1984).  In  addition  to  practice-ori- 
ented AAMRSs,  automated  clinical  research  sys- 
tems have  been  developed  that  also  capture  ambu- 
latory medical  records.  Of  particular  interest  to  this 
study  of  AAMRSs  is  ARAMIS,  developed  by  Fries 
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and  others  at  the  Stanford  University  Medical  Cen- 
ter in  the  early  1970s.  The  system  was  initiated  on 
the  premise  that  solutions  to  problems  of  chronic 
diseases  require  large  data  bases  of  high  quality 
longitudinal  data  to  support  studies  of  disease 
progress  and  outcome  (Shortliffe  and  Barnett, 
1990).  Although  focused  on  arthritis  and  rheuma- 
tology, ARAMIS  provides  access  to  over  22,000 
patients  in  17  institutions. 

Since  the  introduction  of  university-based  auto- 
mated medical  records  systems,  systems  have  been 
developed  that  share  many  of  the  same  characteris- 
tics as  the  original  systems.  These  include  outpatient 
systems  under  development  or  recently  in  prototype 
at  Columbia-Presbyterian  Medical  Center  (CPMS) 
(Shea,  Clark,  and  Clayton,  1990);  Brigham  and 
Women  s  Hospital  in  Boston  (Teich,  Geisler,  Cimer- 
man,  and  others,  1990);  Albany  Medical  Center  in 
Albany  (Ellis,  1990);  Loyola  University  Medical 
Center  in  Chicago  (Price,  Chandrasekhar,  and 
Tamirisa,  1990);  and  University  of  California  at  Los 
Angeles  (UCLA)  (Wilton,  1990).  The  Mayo  Clinic 
is  also  in  the  process  of  automating  its  medical 
records  (Pryor,  Calif f,  Harrell,  and  others,  1985).  In 
addition,  there  are  provider-specific  systems  such  as 
a  family  practice  system  using  PARADOX  and 
TORNADO  (Davis,  1989). 

Ambulatory  health  care  units  are  increasingly 
being  aggregated  in  large  outpatient  clinics,  HMOs, 
and  hospitals,  and  these  medical  records  systems 
might  be  the  appropriate  focus  of  a  combined  data 
base.  Individual  practice  systems  are  important, 
however,  because  of  their  practice  patterns,  even 
though  their  ability  to  participate  in  a  national  col- 
laborative effort  may  be  limited.  Medical  centers, 
HMOs,  and  other  institutions  using  AAMRSs  have 
research  and  technical  staffs  which  support  data 
retrieval  efforts  that  individual  practices  do  not  have. 

AAMRS  systems  also  are  in  use  in  other  coun- 
tries such  as  DIOGENE  in  Geneva,  Switzerland 
(Scherrer,  Baud,  Hochstrasser,  and  Ratib,  1990); 
BAZIS  in  The  Netherlands  (Safran,  1990); 
PEN&PAD,  a  physician  workstation  in  England 
(Nowlan,  Rector,  Kay,  and  others,  1990);  MED- 
SUM,  a  general  practice  medical  record  summary  in 
Australia  (Bridges- Webb,  1986);  a  system  using 
Revelation,  a  personal  computer  data  base  manage- 
ment system,  in  Hong  Kong  (Chan,  Donnan,  Chan, 
and  Chow,  1987);  the  Exmouth  Project  in  England 
that  uses  Smart-Cards  (Dick  and  Steen,  1991);  and 
other  systems  in  The  Netherlands,  Germany,  and 


England.  Lack  of  the  ability  to  access  data  from  for- 
eign countries,  however,  may  be  a  major  barrier  to 
considering  them  for  an  AAMRS  data  base. 

Interstudy,  under  the  direction  of  Paul  Ellwood 
(1988),  has  an  Outcomes  Management  System  that 
includes  survey  instruments  to  describe  patient 
demographics,  functional  status,  treatment,  lifestyle, 
insurance,  and  outcome.  Participating  HMOs  and 
other  health  care  organizations  collect  the  data  for 
quality  assurance  monitoring.  This  system  may  be  a 
source  of  data  for  an  AAMRS  data  base. 

Large  HMOs  such  as  the  Group  Health  Cooper- 
ative of  Puget  Sound  in  Seattle  have  pieces  of  an 
automated  medical  record  system  and  are  consider- 
ing development  of  an  integrated  system.  In  the 
early  1980s,  Kaiser  Permanente  had  an  integrated 
system  under  development,  but  efforts  have  been 
postponed  because  of  funding  (Levinson  and 
Dambro,  1984).  Recent  communication  with  Kaiser 
staff  indicates  that  although  an  integrated  medical 
record  system  does  not  yet  exist,  there  is  hope  that 
development  plans  will  be  reactivated.  Lovelace 
Medical  Center  in  Albuquerque,  which  has  an 
HMO  associated  with  the  hospital  as  well  as  a 
research  foundation,  acknowledges  the  potential 
richness  of  HMO  data  bases  and  has  implemented  a 
Lovelace  Patient  Database  to  support  clinical  stud- 
ies (Lapham,  Montgomery,  and  Hoy,  1990). 

Also  to  be  considered  are  AAMRSs  sponsored 
by  the  Federal  Government.  The  Department  of 
Veterans  Affairs  has  a  system  called  GRAMPS  that 
supports  geriatric  outpatient  practice  (Hammond, 
Prather,  Date,  and  King,  1990)  and  is  considering 
the  development  of  an  AAMRS.  The  Department 
of  Defense  is  implementing  the  Composite  Health 
Care  System  (CHCS)  at  installations  around  the 
world  (Dick  and  Steen,  1991).  The  Research  and 
Patient  Management  System  Center  (RPMS)  is 
used  by  the  Indian  Health  Service  to  service  Indian 
reservations. 

Data  Retrieval 

The  AAMRSs  that  have  been  in  operation  for  a 
length  of  time  such  as  TMR,  COSTAR,  HELP, 
RMRS,  the  Beth  Israel  System,  STOR,  and 
THERESA  have  retrieval  capabilities  that  facili- 
tate analysis  of  the  housed  data.  Adams  (1986) 
reviewed  the  query  languages  that  support 
COSTAR,  RMRS,  and  HELP  and  found  that  they 
allow  nonprogrammers  to  retrieve  data.  It  should 
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be  noted  that  the  application  that  Adams  studied 
supported  patient  care  rather  than  research.  There 
are  numerous  studies  that  have  been  accomplished 
from  these  data  systems,  however,  that  indicate  that 
the  data  can  be  transmitted  to  statistical  packages 
for  research  studies. 

Data  quality.  When  an  existing  AAMRS  is  con- 
sidered for  research,  the  quality  of  the  data  must  be 
reviewed  (Gardiner,  1978).  A  recent  study  (Ellis, 
Krogh,  and  Werth,  1990)  reported  that  when  charts 
were  compared  with  computer  records,  differences 
in  over  47  percent  of  the  diagnoses  were  found.  It 
should  be  noted  that  the  computerized  records  were 
housed  in  a  billing  system  and  not  routinely  used  for 
care.  Their  analyses,  however,  showed  that  differ- 
ences were  attributed  to:  (1)  informal  and  undocu- 
mented coding  rules,  or  lack  of  guidelines;  (2)  soft- 
ware bugs;  and  (3)  difficulty  in  medical  coding.  These 
issues  must  be  addressed  for  AAMRSs  as  well. 

A  review  of  existing  AAMRSs  shows  that  all  of 
the  systems  contain  editing  and  logic  modules  which 
check  relationships  within  and  among  data  ele- 
ments. It  is  clear  that  data  quality  is  an  important 
consideration  for  the  existing  systems  and  that 
within  a  system  confidence  in  the  data  is  high. 

Data  coding  and  standards.  It  has  been  argued 
that  one  of  the  benefits  of  an  automated  ambulatory 
medical  record  is  the  structure  that  it  imposes  on  a 
nonstructured  report.  It  also  has  been  argued,  how- 
ever, that  physicians  resist  structure  and  coding  and 
prefer  to  report  findings  in  the  traditional  narrative 
format.  There  have  been  some  efforts  to  translate 
narrative  reports  into  a  structured  medical  record 
using  linguistic  analyses  (Hirschman,  Story,  Marsh, 
and  others,  1981),  but  this  approach  is  not  used 
widely.  Instead,  the  existing  AAMRSs  use  coded 
terms  and  data  dictionaries  of  medical  terms  particu- 
lar to  each  system.  For  data  to  be  shared  both  within 
and  across  care  settings,  there  must  be  standards  for 
recording  data  (Medical  ADP  systems,  1991). 

Numerous  coding  systems  can  be  found  in  med- 
ical records  generally  and  ambulatory  medical 
records  specifically.  These  include  SNOP  (System- 
atized Nomenclature  of  Pathology),  ICD-9-CM 
(International  Classification  of  Diseases,  9th  Revi- 
sion, Clinical  Modification),  CPT-4  (Current  Proce- 
dural Terminology,  4th  edition),  SNOMED  (System- 
atized Nomenclature  of  Medicine),  MeSH  (National 
Library  of  Medicine's  medical  subject  headings, 
POR  (Problem-Oriented  Record),  and  DRGs 


(diagnosis-related  groups).  These  coding  systems  are 
used  for  different  purposes.  The  ICD-9-CM  system 
is  used  for  diagnoses  and  cause  of  death  categoriza- 
tion, the  CPT-4  for  procedures  categorization,  and 
DRGs  for  reimbursement.  There  have  been  efforts 
to  retrieve  data  using  SNOP  syntax  (Dunham, 
Pacak,  and  Pratt,  1978)  and  to  develop  DRGs  rele- 
vant to  ambulatory  care  settings  (Rogerson,  Stim- 
son,  Simborg,  and  Charles,  1985). 

Feinstein  (1988)  argued  that  a  taxonomy  of  clas- 
sification has  three  requirements:  (1)  suitable  orga- 
nizing principle,  (2)  labeled  categories  with  stand- 
ard titles,  and  (3)  operational  criteria  for  each 
member  of  a  category.  He  pointed  out  that  the 
ICD-9  contains  few  operational  criteria,  and  that 
the  POR  may  be  useful  for  organizing  information 
for  a  single  patient  but  is  idiosyncratic  for  research 
applications. 

The  IOM  report  (Dick  and  Steen,  1991)  sug- 
gested that  in  order  to  develop  a  national  computer- 
ized medical  record  data  system,  standards  are 
required.  Further,  AHCPR  (1991)  noted  that  stand- 
ard medical  terminology  is  required  for  obtaining 
uniform  clinical  data  for  the  Medical  Treatment 
Effectiveness  Program  and  other  health  services 
research. 

Several  efforts  are  underway  to  address  the 
diverse  coding  systems.  The  American  Society  for 
Testing  and  Materials  (ASTM)  is  working  on  the 
standardization  of  medical  terminology  (Gabrieli, 
1990).  The  National  Library  of  Medicine  is  under- 
taking the  development  of  the  Unified  Medical  Lan- 
guage System  (UMLS)  (Lindberg  and  Humphreys, 
1990),  which  is  designed  to  facilitate  the  retrieval 
and  integration  of  information  from  many  coding 
systems  and  sources  of  information  such  as  litera- 
ture, clinical  records,  medical  data  bases,  and  expert 
systems.  It  is  establishing  a  conceptual  link  between 
the  user's  expression  of  an  individual  need  and  the 
relationship  of  information  from  various  machine- 
readable  sources  through  a  metathesaurus.  Colum- 
bia-Presbyterian Hospital  is  augmenting  the  UMLS 
project  by  attempting  to  translate  and  integrate  the 
UMLS  vocabulary  with  its  local  vocabulary 
(Cimino,  Hripcsak,  Johnson,  and  others,  1990). 
Other  projects  are  developing  vocabularies  with  sys- 
tematic, modular  linkages  among  terms  (Rothwell 
and  Cote,  1990). 

Efforts  also  are  underway  in  the  United  King- 
dom and  Europe  to  integrate  terminology 
(Bernauer,  1990;  Rector,  Nowlan,  and  Kay,  1990; 
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Rossi-Mori,  Thornton,  and  Gangemi,  1990).  In 
addition,  the  Arden  Syntax  is  being  used  to  link 
information  between  the  HELP  and  Regenstrief 
systems  (Hripcsak,  Clayton,  Pryor,  and  others, 
1990).  Standard  access  to  computer-based  medical 
resources  is  also  being  approached  through  an  inter- 
active query  workstation  (Cimino  and  Barnett, 
1990a).  The  latter  two  efforts  may  be  applicable  to 
the  trans-data  base  communication  that  is  required 
for  an  AAMRS  data  base. 

Another  related  effort  that  may  be  relevant  for 
an  AAMRS  is  one  that  abstracts  data  from  clinical 
and  hospital  data  bases  for  research  (Thompson, 
Piland,  Hoy,  and  others,  1990).  A  structured 
approach  was  used,  redundant  data  eliminated,  and 
a  relational  data  base  formed.  Although  the 
approach  worked,  the  authors  argued  that  specifica- 
tions of  a  minimum  data  set  would  be  most  useful 
for  research. 

Data  Transmission  Standards 

The  medical  record  is  an  assemblage  of  informa- 
tion from  diverse  sources  such  as  laboratory,  phar- 
macy, pathology,  and  radiology  services.  Although 
many  of  these  ancillary  services  store  information 
electronically,  data  are  not  available  to  the  medical 
record  because  of  the  idiosyncratic  representation 
of  data,  storage  structures,  diverse  record  identi- 
fiers, and  unique  coding  structures.  The  lack  of  data 
compatibility  applies  within  care  settings  and  across 
care  settings.  These  issues  must  be  addressed  in  the 
development  of  standards  for  data  transfer  among 
computer  systems  (McDonald,  1990).  Standards  for 
data  transfer  specify  the  syntax  of  the  message 
including  content  and  format. 

Several  efforts  are  underway  to  develop  stand- 
ards for  data  transfer.  In  the  United  States,  the 
ASTM,  American  College  of  Radiology  and 
National  Electrical  Manufacturers  Association 
(ACR/NEMA),  the  Institute  of  Electrical  and  Elec- 
tronic Engineers  (IEEE),  and  Health  Level  Seven 
(HL7)  are  implementing  efforts  to  develop  stand- 
ards (AHCPR,  1991).  HL7  is  a  consortium  of  ven- 
dors, users,  and  consultants  that  is  developing  inter- 
change standards  for  all  transactions  in  large  medical 
institutions.  It  is  in  place  in  40  sites,  and  these  efforts 
are  closely  allied  with  ASTM.  IEEE's  project,  the 
Medical  Information  Bus  (MIB),  seeks  to  give  inde- 
pendent life  to  data.  A  workstation  also  is  being 


used  to  provide  a  universal  interface  for  hospital 
information  systems  (Shultz  and  Brown,  1989). 

MEDIX  (Medical  Data  Interchange)  is  a  process 
by  which  regional  groups  of  end-users  and  vendors 
develop  inter-  and  intracenter  communications 
through  due  process.  IEEE's  P1175  is  a  MEDIX 
project  (Harrington,  Benson,  and  Spector,  1990). 
Efforts  are  also  underway  in  The  Netherlands 
regarding  standards  through  the  31  and  COPA  proj- 
ects (Branger  and  Duisterhout,  1990;  Hasman, 
Arnou,  Van  Kesteren,  and  Ament,  1990). 

As  the  AHCPR  (1991)  pointed  out,  develop- 
ment of  standards  is  a  first  step  in  the  process  to 
incorporate  standards  into  hardware  and  software 
systems,  demonstrate  the  benefits,  and  gain  wide- 
spread acceptance  in  the  medical  community.  Data 
transmission  standards  are  directly  relevant  to 
determining  the  feasibility  of  an  aggregate  AAMRS 
data  base  to  support  medical  effectiveness  research. 

Implementation  Issues 

Issues  with  respect  to  clinical  data  base  and 
AAMRS  implementation  are  relevant  to  a  collabo- 
rative ambulatory  care  data  base  as  well.  Issues 
include  interface  among  the  various  systems  within 
and  across  health  care  entities  and  the  identification 
of  a  common  core  of  data  elements  to  be  collected. 
McDonald  (1990)  and  others  have  argued  for  clini- 
cal data  interchange  (CDI)  standards.  Another  cur- 
rent issue  is  the  development  of  standards  for  the 
patient  computer-based  record  (PCR)  (Detmer, 
1990;  Dick  and  Steen,  1991)  and  uniform  medical 
languages  (Lindberg  and  Humphreys,  1990).  Issues 
of  privacy  and  confidentiality  as  well  as  data  quality 
continue  to  be  of  concern. 

Barnett  (1988)  commented  that  training  and 
documentation  as  well  as  technical  support  and  pro- 
gramming competence  are  critical  success  factors. 
Pryor,  Califf,  Harrell,  and  others  (1985)  pointed  out 
that  clinical  data  bases  depend  on  stable  funding 
and  a  multidisciplinary  team. 

ARAMIS  provides  a  model  for  a  successful  col- 
laborative data  base  that  can  support  medical  effec- 
tiveness research  efforts.  Developers  of  clinical 
data  bases  stress  several  components  of  successful 
efforts  (Kunitz,  Fishman,  and  Gross,  1982; 
McShane,  Haumann,  and  Glicksman,  1979;  Pryor, 
Califf,  Harrell,  and  others,  1985;  Rosati,  Lee,  Califf, 
and  others,  1982): 
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•  Collaborative,  multidisciplinary  teams. 

•  Broad  research  questions  that  guide  data  collec- 
tion and/or  extraction. 

•  Software  that  facilitates  retrieval  and  analysis. 

•  Ongoing  quality  control. 

Numerous  factors  can  influence  the  practicality, 
utility,  and  viability  of  an  ambulatory  care  data  base. 
Development  and  operational  factors  encompass 
collaboration  and  cooperation,  technology,  method- 
ology, and  costs.  Additional  operational  factors 
include  mode  of  maintenance,  mechanism  for  sup- 
port, and  clinical  site  reimbursement. 

Another  area  requiring  collaboration  is  the  iden- 
tification or  development  of  a  method  for  data 
extraction  and  transmission  to  a  central  data  base 
resource.  Theoretically  and  practically,  data  do  not 
need  to  be  extracted  with  the  same  tool.  Here  again, 
the  issue  is  that  the  same  data  be  extracted  across 
automated  ambulatory  medical  records  systems  so 
that  the  clinical  courses  and  outcomes  of  large  num- 
bers of  patients  can  be  studied.  A  related  problem  is 
that  of  maintaining  patients'  confidentiality.  Individ- 
ual clinical  sites  maintain  name,  address,  and  social 
security  numbers  as  identifiers.  A  method  for 
assigning  patient  numbers  that  are  unique  to  the 
patient  and  follow  the  patient  across  treatment 
encounters  is  needed.  . This  number  must  be  main- 
tained in  the  central  data  base  so  that  data  can  be 
added  to  the  correct  patient  record  over  time. 
Again,  there  are  models  from  existing  data  bases 
that  can  be  applied. 

Funding  and  support  for  ongoing  operation  of  a 
common  data  base  are  concerns  that  cut  across  clin- 
ical data  bases.  Development  costs  are  circum- 
scribed and  have  been  paid  for  by  Federal  grants 
and  contracts  on  numerous  occasions.  Operations 
and  maintenance  also  have  a  history  of  Federal 
funding  (for  example,  the  Framingham  Study,  the 
Cancer  SEER  project,  numerous  long-term  clinical 
trials).  Clinical  data  bases  per  se,  however,  have  not 
had  a  continuing  source  of  funding  (Ellwood,  1988). 
One  of  the  feasibility  tasks  will  be  to  estimate  the 
costs  for  development,  implementation,  and  opera- 
tion of  an  ambulatory  data  base  and  to  identify 
potential  funding  sources. 


Conclusion 

Some  academic  health  centers  have  developed 
automated  ambulatory  medical  records  systems, 
and  these  systems  provide  a  universe  for  potential 
aggregation.  Although  these  systems  are  unique  to 
each  academic  center,  studies  have  been  initiated  to 
extract  like  data  from  a  few  of  the  systems. 

Changes  in  the  health  care  and  reimbursement 
systems,  technical  advances,  and  interest  in  medical 
effectiveness  research  have  caused  numerous 
groups  to  explore  the  possibility  of  developing 
standards  for  a  computerized  medical  record,  to 
develop  standard  medical  vocabularies  and  methods 
for  translating  among  vocabularies,  and  to  develop 
standards  for  data  transmission.  These  efforts  may 
be  facilitated  by  the  development  of  a  central  ambu- 
latory data  base,  and  conversely  the  implementation 
of  a  central  data  base  may  be  facilitated  by  these 
developments. 
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2  m  Medical  Effectiveness  and  Health  Services  Research 


Introduction 

Health  services  research  can  be  classified  as 
methodological,  descriptive,  analytical,  or  experi- 
mental. Research  can  take  place  at  two  levels:  the 
patient-provider  level  and  the  policy  level  (Brook, 
1989).  Patient-provider  level  research  deals  with  the 
appropriateness  or  outcomes  of  care,  and  policy 
research  may  focus  on  levels  of  health  care  expendi- 
tures and  equitable  distribution  of  services.  Models 
to  relate  the  two  are  being  developed. 

Medical  effectiveness  research  "investigates  the 
effects  of  alternative  treatments  on  outcome  real- 
ized by  the  patient"  (Agency  for  Health  Care  Policy 
and  Research  [AHCPR],  1991).  Representatives  of 
Government,  private  insurers,  major  corporations, 
community  agencies,  and  the  medical  profession 
agree  that  in  order  to  provide  a  basis  for  decisions 
on  the  future  funding  and  organization  of  health 
care,  it  is  essential  to  gain  knowledge  about  varia- 
tions in  performance,  relative  costs,  safety,  and 
effectiveness  through  research  (AHCPR,  1991). 

Medical  effectiveness  research  has  been  conduc- 
ted by  means  of  observational  studies  and  controlled 
clinical  trials,  both  of  which  have  their  shortcomings. 
In  an  observational  study,  normal  clinical  practice  is 
recorded  and  outcome  is  compared  among  different 
procedures  or  treatments.  Results  may  be  biased, 
however,  by  systematic  differences  among  patients 
receiving  care.  Often,  observational  studies  (dis- 
cussed in  greater  detail  in  Chapter  5:  Methodological 
Issues)  are  retrospective  and  rely  on  secondary  data 
obtained  from  heterogeneous  sources. 

In  some  cases,  data  used  for  medical  effective- 
ness research  are  administrative  in  nature  and 
include  claims  data  and  association-specific  data 
(for  example,  the  American  Hospital  Association). 
The  amount  of  clinical  data  may  not  be  sufficient  to 
address  the  research  questions.  Severity,  comorbid- 
ity, referral  bias,  or  patient  outcomes  may  not  be 
detailed.  The  validity  of  the  data  and  their  com- 
pleteness and  accuracy  are  often  unknown. 

Controlled  clinical  trials  involve  carefully 
selected  patients  assigned  to  a  treatment  or  proce- 
dure group.  Because  of  the  specificity  of  selected 
patients  and  interventions,  however,  the  results  may 
not  be  generalizable  to  a  broader  population. 


Aspects  of  Medical  Effectiveness  Study 

According  to  Berwick  (1989),  there  are  four  key 
aspects  in  medical  effectiveness  research: 

•  The  study  of  efficacy  (knowing  what  could 
work). 

•  The  study  of  appropriateness  of  care  or  effec- 
tiveness (using  what  works). 

•  The  study  of  execution  of  care  (performing  care 
well). 

•  The  study  of  the  purposes  of  care  (values  that 
underlie  action). 

Berwick  challenged  the  health  care  system  to 
define  the  mission  of  health  care  in  order  to  know 
which  values  guide  the  definition  of  efficacy,  appro- 
priateness, and  quality.  Although  other  authors 
would  define  or  expand  these  aspects  to  include 
variations  in  care  and  outcomes,  as  well  as  other 
factors,  there  is  general  consensus  as  to  the  ele- 
ments involved  in  health  services  research. 

Efficacy 

The  definition  of  efficacy  can  include  diagnostic 
efficacy  to  arrive  at  the  correct  diagnosis,  manage- 
ment efficacy  that  reflects  appropriate  management 
practices,  and  prognostic  or  outcome  efficacy  to 
favorably  improve  patient  outcomes  (Pry or  and 
Lee,  1991).  Systematic  evaluation  of  links  that  con- 
nect a  patient's  actual  condition  to  the  selection  of  a 
diagnostic  test  or  treatment  or  a  combination  of 
these  is  essential  to  determining  efficacy.  This  has 
not  been  carried  out  (Berwick,  1989;  Brook  and 
Lohr,  1985);  however,  technology  assessment 
(Fineberg,  1985),  decision  analysis  (Pauker  and 
Kassirer,  1987),  and  randomized  trials  can  con- 
tribute to  this  evaluation  if  researchers  collaborate 
with  statisticians,  economists,  computer  scientists, 
and  clinicians.  Research  in  efficacy  belongs  at  the 
highest  level  of  aggregation  through  the  Federal 
Government  and  professional  and  scientific  organi- 
zations. The  individual  practitioner,  however,  must 
cooperate  by  sharing  information  and  collaborating 
on  the  use  of  functional  status  measurements  in  rou- 
tine practice  (Nelson  and  Berwick,  1989). 
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Effectiveness  of  Medical  Practice 

Effectiveness,  or  the  efficacy  of  medical  inter- 
ventions in  actual  practice,  and  the  appropriateness 
of  an  intervention  in  a  given  setting  are  important 
concepts  in  health  services  research  (Roper, 
Winkenwerder,  Hackbarth,  and  Krakauer,  1988). 
Knowing  the  efficacy  of  a  particular  treatment  or 
procedure  does  not  translate  directly  into  guiding 
medical  practice  because  the  realities  of  the  health 
care  world  (for  example,  varying  skills  of  providers) 
may  affect  the  effectiveness  or  appropriateness  of 
care.  Therefore,  efficacy  and  effectiveness  research 
must  go  hand  in  hand.  A  way  to  routinely  collect 
information  about  outcomes  of  tests,  procedures, 
drugs,  and  other  health  care  services  would  enhance 
knowledge  about  effectiveness.  Study  not  only  of 
services  to  patients  but  also  of  combinations  of  ser- 
vices or  interactions  of  variables  (for  example,  skill 
of  physicians,  region  of  country)  may  shed  more 
light  on  the  effectiveness  of  various  treatments  and 
the  appropriateness  of  care  (Brook  and  Lohr,  1985). 

Guidelines  to  determine  appropriateness  of  care 
have  been  proposed.  A  review  of  4,500  hospital 
records  showed  that  as  many  as  50  percent  of  the 
cases  studied  did  not  meet  the  criteria  for  appropri- 
ateness of  care  (Chassin,  Kosecoff,  Park,  and  others, 
1987).  For  example,  investigators  found  that  one- 
sixth  of  patients  undergoing  coronary  angiography 
and  upper  gastrointestinal  endoscopy  and  one-third 
of  those  undergoing  carotid  endarterectomy  had 
procedures  deemed  inappropriate  by  a  panel  of 
experts.  Chassin  (1988)  suggested  that  guidelines  for 
specific  diagnostic  tests  or  treatments  that  should  or 
should  not  be  performed  in  certain  circumstances 
should  be  established.  He,  along  with  Brook  at  the 
RAND  Corporation,  used  a  panel  of  experts  to 
arrive  at  a  consensus  about  the  appropriateness  of 
various  procedures  and  therapies  (Nash,  1990).  Eddy 
(1990a)  suggested  that  the  method  used  to  develop 
practice  guidelines  should:  (1)  produce  policies  that 
are  accurate  and  the  outcomes  they  are  supposed  to 
produce;  (2)  be  accountable,  enabling  others  to 
review  the  reasoning  behind  the  policy;  (3)  enable 
people  to  anticipate  the  health  and  financial  conse- 
quences of  applying  the  policy;  (4)  facilitate  the  reso- 
lution of  conflicts  across  policies;  and  (5)  facilitate 
the  application  of  the  policy  both  to  individual 
patients  and  to  populations.  The  outcome  should 
determine  the  desirability  of  different  interventions. 

Efforts  to  establish  guidelines  for  clinical  prac- 
tice have  taken  place  in  a  variety  of  arenas.  The 


Harvard  Community  Health  Plan  has  developed 
guidelines  for  the  care  of  common  ambulatory  con- 
ditions. However,  more  comprehensive  data  are 
needed  to  facilitate  guideline  development  and 
enhance  the  credibility  of  guidelines.  In  1990, 
RAND,  the  Academic  Medical  Center  Consortium, 
and  the  American  Medical  Association  agreed  to 
establish  a  Clinical  Appropriateness  Initiative  to 
"develop  criteria,  practice  parameters,  and  other 
processes  to  identify  and  promote  appropriate 
care ..."  (Berman,  Kottke,  and  Ballard,  1990). 

The  Agency  for  Health  Care  Policy  and 
Research  has  funded  patient  outcome  research 
teams  (PORTs)  to  study  specific  common  condi- 
tions and  the  most  effective  treatments  for  them. 
These  teams  document  the  range  of  treatments 
available,  examine  variations  in  use  of  the  treat- 
ments, establish  probabilities  for  outcomes  that  are 
important  to  patients,  and  gain  support  and  collabo- 
ration from  providers  (Wennberg,  1990). 

The  Health  Care  Financing  Administration 
(HCFA)  has  been  involved  in  a  medical  effective- 
ness initiative  utilizing  claims  processing  and  peer 
review  data  to  monitor  trends  and  assess  effective- 
ness of  specific  interventions.  HCFA  views  assess- 
ment of  medical  effectiveness  as  involving  monitor- 
ing, analysis  of  variation  in  practice  patterns  and 
outcomes,  assessment  of  interventions,  and  feed- 
back and  education.  Although  HCFA  data  allow 
patient  characteristics  to  be  linked  with  treatment, 
the  lack  of  clinical  detail  and  specific  information 
limits  the  utility  of  the  data.  In  addition.  Medicare 
and  Medicaid  data  may  not  be  representative  of  the 
general  population  (Roper,  Winkenwerder,  Hack- 
barth, and  Krakauer,  1988). 

Inherent  in  a  discussion  of  effectiveness  are  the 
conceptually  overlapping  issues  related  to  variations 
in  care  and  outcome. 

Variations  in  Care 

Through  the  research  of  Wennberg  (1985a, 
1985b,  1991)  and  Wennberg,  Freeman,  and  Culp 
(1987),  systematic  and  persistent  differences  in  the 
rates  of  use  of  medical  care  and  common  surgical 
procedures  have  been  discerned.  Using  a  methodol- 
ogy borrowed  from  epidemiology  and  known  as 
small  area  analysis,  Wennberg  and  colleagues  pro- 
posed that  variations  in  practice  style  can  be  attrib- 
uted to  two  different  causes  (Wennberg,  1985b). 
First,  controversies  about  efficacy  and  effectiveness 
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of  alternative  therapies  cannot  be  resolved  because 
the  scientific  information  is  inadequate.  Low  varia- 
tion procedures  are  generally  those  for  which  there 
is  professional  consensus  on  diagnosis  and  preferred 
place  or  style  of  treatment.  However,  well-defined 
norms  do  not  exist  for  many  conditions — including 
those  related  to  aging,  hysterectomy,  prostatectomy, 
coronary  bypass,  and  others — and  the  Wennberg 
studies  do  not  provide  conclusions  about  which 
treatment  produces  a  better  outcome.  Patient  and 
provider  data  need  to  be  linked  with  outcome  meas- 
ures to  determine  the  effectiveness  of  treatment 
(Wennberg,  Mulley,  Hanley,  and  others,  1988). 
Therefore,  treatment  may  depend  on  the  experi- 
ence of  the  physician.  This  becomes  the  basis  for  the 
second  cause  of  variation  in  practice  and  is  less 
likely  related  to  scientific  controversy.  Physicians' 
choices  of  treatment  may  be  motivated  by  their  own 
or  their  patients'  preferences  or  convenience  or 
their  perceptions  of  their  need  to  protect  themselves 
from  liability  (Barry,  Mulley,  Fowler,  and 
Wennberg,  1988;  Friedman,  1986). 

Eddy  (1984)  agrees  that  uncertainty  exists  in 
physicians'  information  processing,  history  taking, 
symptom  observation,  and  diagnosis.  A  study  of 
these  aspects  of  clinical  practice  is  necessary  to 
design  better  guidelines  for  treatment  and  care. 
Wennberg  (1985b)  labeled  these  differences  the 
"practice  style  factors"  and  noted  their  effect  on 
medical  effectiveness  research  and  cost  contain- 
ment. For  example,  differences  in  the  need  for  hos- 
pitalization are  the  most  important  determinant  in 
per  capita  costs  of  treatment  for  specific  diseases. 
The  decision  to  treat  in  an  ambulatory  setting  ver- 
sus an  inpatient  setting  affects  the  resources  needed 
and  has  profound  implications  for  the  patient  and 
the  payer.  Differences  are  worldwide  and  are  not 
explained  by  the  incentive  associated  with  fee-for- 
service  medicine. 

A  wide  variety  of  studies  on  variations  in  care 
exist.  For  example,  a  review  of  the  use  of  respira- 
tory therapy  at  a  Massachusetts  hospital  found  it 
far  exceeded  the  state-wide  average  (Zibrak,  Ros- 
setti,  and  Wood,  1986).  When  advised  about  the 
optimal  use  of  various  respiratory  therapy  options 
by  trained  respiratory  therapists,  physicians 
decreased  their  use  of  respiratory  therapy  with  no 
effect  on  morbidity  and  mortality  from  pulmonary 
disorders.  In  two  other  studies,  differences  in  hospi- 
talization rates  of  internists  and  family  physicians 
were  examined.  One  study  (Franks  and  Dickinson, 


1986)  found  no  interspecialty  differences  in  length 
of  stay,  hospital  charges,  number  of  procedures 
done,  or  disposition.  Family  physicians,  however, 
recorded  fewer  diagnoses  than  did  internists.  The 
second  study,  using  the  MedisGroups  comparative 
data  base,  compared  outcome  measures  in  hospital- 
ized patients  over  age  65  treated  by  internists  and 
family  physicians  (McGann  and  Bowman,  1990). 
According  to  this  study,  patients  admitted  by  family 
physicians  had  a  significantly  higher  admission  ill- 
ness severity  and  were  significantly  older;  however, 
there  were  no  differences  in  morbidity  and  mortal- 
ity. The  lower  charges  for  patients  admitted  by  fam- 
ily physicians  were  statistically  significant  and 
reflected  variation  in  the  use  of  diagnostic  tests  and 
procedures.  In  still  another  study  describing  the 
extent  of  variability  in  diagnosis  and  treatment  of 
temporomandibular  disorders,  the  authors  noted 
that  the  differences  were  not  related  solely  to 
patient-reported  pain  and  dysfunction  (Von  Korff, 
Howard,  Truelove,  and  others,  1988).  The  results 
indicate  the  need  for  systematic  approaches  to 
identifying,  evaluating,  and  modifying  variations  in 
health  care  practice. 

Dittus  and  Tierney  (1987),  using  Regenstrief 
data,  examined  variation  in  how  often  physicians 
see  outpatients  for  followup  visits.  They  found  a 
four-fold  difference  in  return  visit  intervals  between 
the  longest  and  shortest  intervals  for  followup.  Dif- 
ferences could  not  be  accounted  for  by  patients' 
diagnosis  or  clinical  status.  In  addition,  physicians 
with  the  shortest  followup  interval  ordered  more 
diagnostic  tests. 

Wennberg  and  colleagues,  in  responding  to  their 
and  others'  observations  of  variations  in  care,  sug- 
gested that  compiling  and  distributing  reports 
detailing  per  capita  use  rates  for  specific  health  care 
markets  will  assist  in  addressing  variations  in  care 
(Wennberg,  Freeman,  and  Gulp,  1987).  These 
reports  would  need  to  be  based  on  systematically 
collected,  accurate  data  that  are  available  across 
market  areas.  Health  information  about  total  popu- 
lations would  enable  better  decisionmaking  and 
planning  in  the  health  care  field  (Wennberg  and 
Gittelsohn,  1973).  Closer  monitoring  of  medical 
practice  could  obtain  population-based  measures  of 
resource  allocation,  service  use,  and  outcomes  of 
care.  Health  insurance  records  (Medicare,  private 
insurers)  and  hospital  discharge  abstracts  contain 
information  on  hospital  resources  used.  Discharge 
and  claims  records  establish  the  link  between  use  of 
medical  care,  diagnosis,  and  outcome.  Hospital 
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outpatient  data,  emergency  room  data,  and  physi- 
cians' office  data  would  present  a  view  of  the  use  of 
ambulatory  care.  Geographic  location  and  organiza- 
tion of  the  health  care  community  may  influence 
resource  allocation  and  service  use  rates. 

The  Wennberg  reports  would  aid  in  addressing 
the  effectiveness  of  common  therapeutic  interven- 
tions. The  feedback  of  findings  on  practice  varia- 
tions and  outcomes  to  State  medical  associations, 
specialty  societies,  and  individual  providers  would 
enable  them  to  reconsider  indications  for  specific 
services  and  reduce  the  variation  in  practice  pat- 
terns. Cost  containment  could  be  achieved  by  guide- 
lines addressing  the  use  of  outpatient,  ambulatory 
services  in  place  of  inpatient  care.  Reduction  in  hos- 
pitalization should  not  negatively  affect  the  patient- 
physician  relationship  nor  should  it  have  a  negative 
economic  impact  on  physicians.  Although  this 
approach  may  raise  concern  about  the  effect  on  out- 
come, little  or  no  information  exists  to  confirm  this 
fear.  In  a  study  comparing  use  of  hospital  beds  in 
New  Haven  and  Boston,  Wennberg,  Freeman,  and 
Culp  (1987)  found  no  discernible  difference  in  out- 
come but  did  find  a  great  difference  in  hospitaliza- 
tion rates  for  conditions  for  which  hospital  admis- 
sion is  discretionary  (for  example,  back  problems, 
gastroenteritis,  simple  pneumonia).  In  a  similar 
study  (Perrin,  Homer,  Berwick,  and  others,  1989), 
hospitalization  rates  for  children  in  Boston, 
Rochester,  and  New  Haven  were  shown  to  vary 
greatly.  No  medical  reasons  for  variations  were 
proven,  but  socioeconomic  status  and  access  to  pri- 
mary care  may  have  contributed  to  differences.  The 
implications  for  cost  and  quality  of  care  are  obvious. 

In  Great  Britain,  a  number  of  evaluation  tech- 
niques and  cost  benefit  studies  have  been  imple- 
mented to  assist  in  decisions  about  the  relative  ben- 
efits of  various  interventions  in  the  treatment  of  car- 
diovascular conditions.  The  decision  as  to  whether 
costs  are  worthwhile  (that  is,  different  use  of  tech- 
nology or  other  interventions  enhances  outcome) 
will  be  included  in  the  choice  of  the  intervention 
(O'Brien  and  Rushby,  1990). 

Another  aspect  of  variation  in  care  addresses  the 
provider  of  care.  Data  drawn  from  the  Columbia 
Medical  Plan  were  used  to  evaluate  the  relative  cost 
benefit  of  physicians  and  nurse  practitioners  in  a 
pediatric  setting  (Salkever,  Skinner,  Steinwachs,  and 
Katz,  1982).  Results  indicated  that  although  nurse 
practitioners  are  less  costly  than  physicians  in  pro- 
viding care,  they  are  not  less  effective. 


Outcome  Measures 

Well-defined  and  reliable  outcome  measures 
allow  researchers  to  study  disease  natural  history, 
clinical  course,  and/or  the  effects  of  treatment  inter- 
ventions on  similar  groups  of  patients.  In  recent 
years,  there  has  been  an  increase  in  the  range  of 
available  outcome  measures.  Until  the  1970s,  most 
clinical  studies  defined  outcome  as  a  dichotomous 
variable,  "dead"  or  "alive."  New  and  better  treat- 
ments, increased  longevity  of  the  population,  and  a 
growing  focus  on  chronic  disease  created  the  need 
for  more  encompassing  measures  of  outcome. 
Improved  technology  yielding  a  choice  of  treat- 
ments, increased  competition  among  health 
providers,  and  increased  consumer  awareness  have 
resulted  in  the  need  to  include  patient  satisfaction 
and  quality  of  life  as  outcome  dimensions.  Financial 
constraints  have  influenced  the  inclusion  of  effi- 
ciency and  cost-effectiveness.  The  interest  in  the 
past  5  years  on  quality  of  care  and  quality  assurance 
has  also  led  to  an  increased  interest  in  outcome. 

Outcome  measurement  is  a  fertile  area  of 
research  with  much  discussion  as  to  the  domains 
encompassed,  mechanisms  for  measurement,  mode 
of  administration  (health  care  practitioner  or  self), 
and  the  measures.  Also  actively  debated  is  whether 
there  should  be  a  global  outcome  measure  or  dis- 
ease- and/or  age-specific  measures. 

In  an  effort  to  give  patients,  providers,  and  reim- 
bursers  better  insight  into  making  care-related 
choices  and  decreasing  variations  in  care,  Ellwood 
(1988)  proposed  an  outcomes  management 
approach  to  care.  In  the  Shattuck  lecture,  he  sug- 
gested that  a  national  data  base  containing  informa- 
tion on  clinical,  financial,  and  health  outcomes  be 
built  so  that  the  relationship  between  medical  inter- 
ventions, health  outcomes,  and  costs  can  be  esti- 
mated. Wennberg  (1985b)  suggested  that  a  plan  be 
designed  and  implemented  to  explain  whether  a 
specific  intervention  to  treat  a  particular  condition 
with  a  known  level  of  morbidity  gets  a  better  result 
than  another  treatment. 

Outcomes  management  would  place  greater 
reliance  on  guidelines  and  standards  for  use  of  inter- 
ventions, measure  the  functioning  and  well-being  of 
patients,  pool  clinical  and  outcome  data  on  a  large 
scale,  and  analyze  and  disseminate  results  into  prac- 
tice. It  would,  among  other  things,  generate  an 
understanding  of  the  variations  in  practice  style 
among  providers  in  different  geographic  locations. 
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Research  on  outcomes  is  important  in  studies  of 
the  effectiveness  of  medical  practice  and  may 
increase  the  efficiency  of  monitoring  care  (Epstein, 
1990).  For  the  majority  of  patients,  outcome  meas- 
ures related  to  reduction  of  symptoms,  improve- 
ment in  daily  functioning,  or  improvement  in  a 
sense  of  well-being  and  health-related  quality  of  life 
are  more  appropriate  than  death  as  an  outcome 
(Greenfield,  Kaplan,  and  Ware,  1985).  Methods 
used  in  outcomes  research  allow  for  heterogeneity 
in  patients.  Case  mix  adjustment  separates  the 
effects  of  care  from  those  of  preexisting  health  sta- 
tus and  other  influencing  factors.  Many  attempts  are 
being  made  to  develop  a  severity  of  illness  index 
(Horn  and  Horn,  1986;  Iezzoni,  1990),  distinguish 
costs  due  to  severity  differences  across  patients,  and 
assist  in  identifying  issues  related  to  quality  of  care 
and  outcome  (Iezzoni,  1990).  According  to  a  study 
of  mortality  from  transurethral  resection  of  the 
prostate,  case  mix  does  not  explain  the  difference  in 
outcome  through  its  influence  on  the  choice  of 
treatment  (Roos,  Wennberg,  Malenka,  and  others, 
1989).  Further  research  in  this  area  is  needed. 

There  is  interest  in  measures  of  physical  and 
mental  health,  social  and  role  functioning,  and 
other  general  health  concepts  to  be  used  in  evaluat- 
ing outcome.  To  be  useful,  these  measures  should 
address  a  variety  of  health  concepts  and  health 
states  and  should  adhere  to  standards  of  reliability 
and  validity  (Stewart,  Hays,  and  Ware,  1988). 

A  variety  of  approaches  to  measuring  functional 
outcomes  have  been  developed  and  are  being 
refined.  The  Sickness  Impact  Profile  and  the  Index 
of  Well-being  are  sound  psychometric  measures.  The 
McMaster  Health  Index  Questionnaire,  the  Duke- 
University  of  North  Carolina  Health  Profile  (Parker- 
son,  Broadhead,  and  Tse,  1990),  and  the  RAND 
Health  Insurance  Experiment  (HIE)  (Brook,  Kam- 
berg,  Lohr,  and  others,  1990)  provide  accurate  meas- 
ures of  outcome.  Further  research  is  needed  to 
determine  the  practicality  of  these  instruments  for 
use  in  clinical  settings. 

Other  surveys  have  been  developed  based  on  a 
few  single-item  measures.  For  example,  Spitzer, 
Dobson,  Hall,  and  others  (1981)  developed  a  qual- 
ity-of-life  index  that  aggregates  five  single-item 
measures  of  health  and  health-related  concepts  in 
a  questionnaire  that  takes  1  minute  to  administer. 
However,  single-item  measures  often  are  less  reli- 
able and  less  valid  than  longer  measures  (Stewart, 
Hays,  and  Ware,  1988).  As  a  compromise  between 


long  and  short  instruments,  Stewart  and  her  col- 
leagues developed  a  general,  short  health  survey  of 
function  and  well-being  that  is  reliable,  self-admin- 
istered, and  comprehensive.  Roles,  social  interac- 
tions, physical  function,  emotions,  and  perceptions 
of  health  and  pain  are  addressed  in  a  5-minute 
questionnaire. 

The  Medical  Outcome  Study  (MOS)  was 
designed  to  determine  whether  variations  in  patient 
outcomes  are  explained  by  differences  in  the  system 
of  care,  clinician  specialty,  or  clinician  technical  or 
interpersonal  style  and  to  develop  a  more  practical 
tool  for  monitoring  patient  outcomes  in  medical 
practice.  The  MOS  was  implemented  in  Boston, 
Chicago,  and  Los  Angeles  in  areas  where  a  large 
health  maintenance  organization  (HMO)  and  physi- 
cian specialty  groups  were  willing  to  participate  in 
the  study.  Information  about  patient  case  mix,  varia- 
tion in  technical  and  interpersonal  style  of  practice, 
patient  and  provider  characteristics,  patient  function, 
and  other  outcomes  was  collected.  Not  all  of  this 
information  is  generally  noted  in  medical  records, 
claims  forms,  or  other  available  sources.  Data  collec- 
tion from  clinicians,  patients,  medical  records,  and 
independent  clinical  examination  was  undertaken 
(Tarlov,  Ware,  Greenfield,  and  others,  1989).  Ware's 
(1991)  interest  in  the  patient's  perspective  on  out- 
come of  medical  care  led  him  to  combine  traditional 
clinical  measures  of  disease  status  with  the  patient's 
view  of  disease  and  treatment.  He  suggested  that  it  is 
not  possible  to  move  from  efficacy  research  to  effec- 
tiveness research  unless  new  tools  to  define  effec- 
tiveness include  quality  of  life,  function,  and  well- 
being  from  the  patient's  perspective.  Existing  data 
bases  do  not  include  the  patient's  experience  of 
health  care  and  health  outcome,  and  therefore,  new 
data  bases  need  to  be  developed. 

Today's  technology  allows  storage  and  linkage 
of  numerous  patient  descriptors  with  outcome  and 
provides  the  opportunity  to  develop  and  assess 
more  comprehensive,  sensitive,  cost-  and  time- 
effective  measures.  The  large  numbers  of  patients 
in  a  data  base  and  the  capability  of  analyzing  multi- 
ple independent  and  dependent  variables  should 
aid  in  focusing  on  those  measures  that  are  reliable 
and  most  adequately  describe  outcome.  Studies  of 
outcome  can  be  enhanced  and  models  refined  with 
an  ambulatory  care  research  resource  that  can  link 
symptoms,  treatment,  followup,  and  measures  of 
outcome. 
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Quality  of  Care 

The  study  of  the  execution  of  care  or  quality  of 
care  is  integral  to  effectiveness,  variations  in  care, 
and  outcomes.  Computerized  medical  records  have 
improved  the  ability  to  carry  out  quality  of  care 
research  with  large  numbers  of  patient  records. 
Although  the  tools  for  assessing  quality  of  care  have 
been  developed  over  the  years,  they  have  not  been 
integrated  into  larger  models  of  health  care  delivery 
that  would  enable  researchers  to  understand  the 
meaning  of  variations  in  the  use  of  services  or  to 
identify  acceptable  ranges  of  effectiveness. 

In  1986,  HCFA  released  a  public  report  on  hos- 
pital-specific death  rates  of  Medicare  inpatients  in 
more  than  5,500  American  hospitals  to  reflect  qual- 
ity of  care  in  those  facilities.  Although  HCFA 
attempted  to  define  a  multivariate  case  mix  model 
so  that  severity  of  illness  could  be  accounted  for,  it 
has  been  criticized  for  publishing  misleading  data. 
Although  death  rates  could  reflect  quality  of  care, 
assumptions  inherent  in  the  model  were  questioned 
(Berwick  and  Wald,  1990).  Other  studies  (Hannan, 
Bernard,  O'Donnell,  and  Kilburn,  1989)  also 
attempted  to  target  quality  of  care  problems  by 
reviewing  mortality  data.  At  the  very  least,  interest 
in  understanding  variations  in  quality  of  care  has 
been  raised  by  the  HCFA  study.  In  a  review  of  liter- 
ature on  differences  in  hospital  mortality,  Fink, 
Yano,  and  Brook  (1989)  noted  that  hospital  mortal- 
ity data  were  not  gathered  uniformly,  were  collected 
for  only  a  short  time,  and  often  came  only  from 
local  hospitals.  Inability  to  measure  differences  in 
patients  was  a  common  concern  of  the  authors  of 
these  articles.  Recommendations  to  improve  studies 
included  acquiring  longitudinal,  prospective  data 
from  a  variety  of  facilities  in  different  geographical 
locations  and  developing  strategies  for  translating 
the  results  into  more  effective  practices. 

Concern  for  quality  of  care  in  the  ambulatory 
setting  has  grown  as  increasingly  more  care  is  being 
provided  there.  In  addition,  there  is  less  peer 
review  or  quality  assurance  in  ambulatory  settings 
than  in  inpatient  settings.  According  to  Brook, 
Kamberg,  Lohr,  and  others  (1990),  most  studies  of 
the  quality  of  ambulatory  care  have  been  narrowly 
focused,  examined  one  disease,  or  involved  patients 
from  a  single  institution  or  geographical  area.  In 
general,  data  in  each  study  came  from  a  single 
source  such  as  insurance  claims  forms  or  medical 


records.  Most  studies  examined  either  the  process 
of  care  or  the  outcomes  of  care,  but  not  both.  When 
outcome  was  assessed,  it  was  for  1  year  or  less.  In 
many  cases,  the  population-based  studies  were  per- 
formed many  years  ago.  The  study  by  Brook  and 
colleagues  (1990)  of  the  quality  of  care  rendered  in 
an  ambulatory  care  setting  for  17  common  chronic 
conditions  was  based  on  data  obtained  from  the 
RAND  HIE.  Quality  of  care  was  measured  by 
process  and  outcome  criteria  to  enhance  the  results 
of  the  study.  Other  researchers  (Fries,  1983a)  found 
that  although  process  assessments  yield  the  "hard 
data,"  such  as  laboratory  results,  outcome  data 
reflect  quality  of  life  and  well-being.  In  the  Brook 
(1990)  study,  outcomes  dealt  with  physiological 
measures  and/or  the  effect  of  the  condition  on  the 
patient.  Process  criteria  included  the  appropriate 
use  of  visits  and  diagnostic  tests  or  the  appropriate- 
ness of  the  interventions.  Data  were  obtained  from 
insurance  claims,  screening  physical  examinations 
at  the  beginning  and  end  of  the  study,  and  patient 
surveys  from  patients  in  six  U.S.  sites  who  were 
randomly  assigned  to  insurance  plans  that  were 
free  or  required  cost  sharing,  or  in  one  site,  to  an 
HMO.  No  data  were  collected  from  medical 
records.  Physiologic  outcome  standards  were 
shown  to  have  been  met  80  percent  of  the  time. 
Standards  for  the  effect  of  the  condition  on  the 
patient  were  found  to  be  met  80  percent  of  the 
time.  Process  of  care  standards  were  met  less  fre- 
quently than  were  outcomes  criteria  (55  percent). 
According  to  Brook  and  coauthors,  there  seems  to 
be  more  room  for  improvement  in  process  of  care 
than  in  outcomes  of  care  in  this  population  with 
chronic  conditions.  This  position,  however,  assumes 
that  a  direct  relationship  exists  between  process 
and  outcome  and  does  not  address  the  relative 
worth  in  dollar  costs  of  improving  the  process  to 
improve  the  outcome.  As  a  result  of  these  findings, 
Brook  and  colleagues  (1990)  recommend  addi- 
tional work  to  link  process  and  outcome  models  of 
the  treatment  of  ambulatory  chronic  conditions  so 
that  quality  assessment  focuses  on  improving 
process,  and  therefore,  outcome  with  no  unneces- 
sary increase  in  the  costs  of  care.  Other  researchers 
do  not  necessarily  agree  with  this  approach. 
Equally  important,  this  study  demonstrated  the 
type  of  research  possible  when  there  is  access  to 
large  amounts  of  longitudinal  data  that  can  address 
issues  of  outcome  and  quality  of  care. 
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Summary 

Medical  effectiveness  research  on  efficacy,  effec- 
tiveness, appropriateness,  and  outcome,  as  well  as 
variations  in  care  and  the  quality  of  care,  requires 
data  on  the  setting  where  medical  care  is  delivered, 
providers  giving  care,  differences  among  patients, 
and  the  process  and  outcomes  of  care  ( AHCPR, 
1990).  The  level  of  detail  needed  to  address  these 
areas  is  not  likely  to  be  found  in  existing  secondary 
data  sources  nor  in  clinical  trial  data.  Primary  data 
collection  can  provide  more  complete  information 
on  a  patient  at  a  given  time  and  over  time.  Linking 
of  these  records  will  increase  the  number  of  individ- 
ual records  available  for  study  and  enhance  the  gen- 
eralizability  of  the  results.  The  use  of  existing  auto- 
mated ambulatory  medical  records  systems  is  one 
mechanism  for  utilizing  primary  clinical  data  in  a 
cost  and  time-effective  manner. 
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■  Decisionmaking 


Introduction 

In  the  process  of  making  a  clinical  decision,  a 
physician  is  faced  with  several  possible  alternatives 
that  must  be  weighed  before  the  physician  makes  a 
decision  (Dannenberg,  Shapiro,  and  Fries,  1979). 
According  to  Dawson  and  Connors  (1990),  physi- 
cian decisionmaking  is  influenced  by  the  following 
factors:  (1)  information  about  diagnosis,  prognosis, 
treatment  alternatives,  and  patient  preferences;  and 
(2)  physician-specific  characteristics,  such  as  person- 
ality, decisionmaking  style,  biases,  confidence,  per- 
sonal preferences,  and  external  pressures.  To  make 
accurate  decisions,  it  is  essential  to  have  accurate 
data,  pertinent  knowledge,  and  appropriate  prob- 
lem-solving skills  (Shortliffe  and  Perreault,  1990). 

The  Decisionmaking  Process 

The  clinical  decisionmaking  process  involves  the 
estimation  and  comparison  of  possible  alternatives 
with  the  outcomes  of  each  of  these  generated  alter- 
natives (Dannenberg,  Shapiro,  and  Fries,  1979).  The 
first  part  of  this  process  is  a  "question  of  facts"  and 
deals  with  data  collection  and  analysis.  The  second 
part  is  a  "question  of  personal  values  or  prefer- 
ences," consisting  of  the  comparisons  between  the 
potential  benefits  and  harms  of  a  given  practice,  the 
health  outcomes  associated  with  a  given  practice 
and  its  related  costs,  and  the  benefit  gained  for  each 
of  the  possible  practices  (Eddy,  1990b).  During  this 
process,  misperceptions  of  the  outcomes  or  of  the 
values  that  patients  place  on  the  outcomes  may 
result  in  flawed  decisions.  The  computer  can  be 
used  to  make  decisions  about  diagnosis  and  process 
of  care.  Process  of  care  may  include  knowing  what 
questions  to  ask,  tests  to  perform,  and  procedures  to 
carry  out.  It  also  may  include  determination  of  the 
value  of  the  results  based  on  risks  to  the  patient  and 
financial  costs. 

Feinstein,  Rubinstein,  and  Ramshaw  (1972)  sug- 
gested that  the  decisionmaking  process  is  hindered 
by  the  following  problems:  (1)  limited  physician 
experience,  (2)  lack  of  direct  access  to  a  population 
of  sufficient  size  for  the  purpose  of  properly  match- 
ing patient  conditions,  and  (3)  the  decisionmaker's 


inability  to  accurately  estimate  therapeutic  out- 
comes. As  a  result,  these  authors  advocated  the 
implementation  of  a  computer  program  that  facili- 
tates the  physician's  estimation  of  outcomes  by  pro- 
viding access  to  a  large  data  base  of  clinical  experi- 
ence and  information.  Computers,  then,  can  be 
valuable  assets  in  decision  support  as  tools  for  infor- 
mation management,  focusing  attention,  and  spe- 
cific patient  data  (Shortliffe  and  Perreault,  1990). 

According  to  Mutschler  (1990),  decision  support 
systems  are  directed  at  unstructured  problems,  use 
decisionmaking  models  and  analytic  techniques,  and 
attempt  to  enhance  the  effectiveness  of  the  decision- 
making process  without  supplanting  the  role  of  the 
decisionmaker.  Using  a  large  and  complex  data  base 
enables  the  physician,  with  the  aid  of  a  decision  sup- 
port system,  to  consider  diagnoses,  prognoses, 
and/or  treatments  that  otherwise  might  have  been 
overlooked  or  inappropriately  discarded.  In  fact,  one 
study  suggested  that  failure  to  make  a  correct  deci- 
sion may  be  due  to  the  failure  of  the  physician  to 
generate  a  comprehensive  list  of  diagnostic  possibili- 
ties that  not  only  contains  the  correct  diagnosis  but 
includes  only  the  more  likely  ones  (Barnett,  Cimino, 
Hupp,  and  Hoffer,  1987;  White,  1985). 

To  obtain  such  a  list,  the  physician  first  must 
record  the  necessary  patient  information  which,  for 
optimum  efficiency,  should  have  diagnostic  signifi- 
cance and  the  greatest  potential  for  reducing  the  set 
of  possible  diagnoses.  In  generating  this  list,  the  user 
must  define  each  observation  correctly  because  the 
inappropriate  use  of  medical  terms  or  inaccurate 
recording  of  physician  observations  will  most  likely 
result  in  incorrect  assessment  of  the  patient's  condi- 
tion (Dietrich,  1978). 

Decision  Support  Systems 

The  various  types  of  clinical  decisionmaking  sys- 
tems can  be  characterized  by  their  attributes.  The 
system  function  may  enable  the  computer  to  pro- 
duce facts  about  the  patient  (diagnosis)  or  may  sug- 
gest what  to  do  for  the  patient  (process  of  care). 
The  mode  of  giving  advice  may  include  identifica- 
tion of  the  diagnosis  and  the  capacity  to  respond  to 
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the  providers1  inquiries  about  the  process  of  caring 
for  a  patient  with  that  diagnosis,  or  it  may  automati- 
cally give  the  provider  the  options  for  care.  The 
consultation  style  may  generate  advice  for  the 
provider  about  diagnosis  and/or  management,  or  it 
may  utilize  a  critiquing  style  in  which  the  computer 
responds  to  the  providers'  suggestions  for  care  by 
agreeing  or  disagreeing.  The  underlying  decision 
science  methodology  may  range  from  simple  logic 
to  more  sophisticated  Bayesian  statistics,  decision 
analysis,  artificial  intelligence,  or  a  combination  of 
these.  The  human-computer  interface  characteris- 
tics include  logistical  considerations  of  access  and 
ease  of  use,  mechanical  considerations  such  as  types 
of  data  entry  (keyboard,  mouse,  touch  screens),  and 
psychological  considerations.  The  combination  of 
these  characteristics  may  determine  the  utility  of  the 
computer  and  its  acceptability  by  health  care 
providers  (Shortliffe  and  Perreault,  1990). 

Decision  support  systems  have  been  created  for 
such  areas  as  glaucoma  (Weiss,  Kulikowski,  and 
Safir,  1978),  lung  cancer  (Feinstein,  Rubinstein,  and 
Ramshaw,  1972),  endocrinology  (Schild,  Lunenfeld, 
and  Gavish,  1978),  cardiology  (Nomura,  1974),  and 
radiology  (Dietrich,  1978).  The  glaucoma  consulta- 
tion system,  for  example,  incorporated  a  network  of 
medical  knowledge  necessary  for  reasoning  and 
decisionmaking  processes  (Weiss,  Kulikowski,  and 
Safir,  1978).  In  simulating  the  physician  s  own  diag- 
nostic process,  the  interactive  system  proposed  by 
Schild,  Lunenfeld,  and  Gavish  (1978)  prompts  the 
physician  for  relevant  patient  data  to  ensure  that 
the  amount  of  information  requested  is  minimal. 

Conclusion 

Although  Barnett,  Cimino,  Hupp,  and  Hoffer 
(1987)  stated  that  information  systems  provided 
better  access  to  knowledge  bases  than  did  textbooks 
or  journal  articles,  they  also  identified  some  prob- 
lems hindering  the  effect  of  such  systems  on  the 
physician  decisionmaking  process:  (1)  difficulty  in 
obtaining  access  and  interacting  with  the  systems, 
(2)  limited  coverage  of  diseases  and  conditions  in 
the  systems,  (3)  limited  ability  of  the  decision  sup- 
port system  to  explain  and  justify  its  interpretations, 
and  (4)  poor  quality  of  these  interpretations. 
Regardless  of  these  limitations,  Dannenberg, 
Shapiro,  and  Fries  (1979)  found  that  computer-gen- 
erated prognoses  resulted  in  an  improved  ability  for 
physicians  to  predict  therapeutic  outcomes. 

There  is  a  dearth  of  information  in  the  literature 
related  to  immediate  access  to  data  bases  that 


would  enable  clinical  decisionmaking.  Issues  related 
to  making  collected  data  available  in  a  timely  man- 
ner are  key  to  enhancing  the  use  of  computer-based 
decisionmaking  in  clinical  practice. 

Implementation  of  automated  ambulatory  med- 
ical records  systems  in  clinical  practice  not  only 
improves  the  physician's  ability  to  process  informa- 
tion and  record  medical  history  and  observations  of 
symptoms  but  also  results  in  better  feedback  con- 
cerning treatment  outcomes  and  enhances  physician 
decisionmaking  ability.  Furthermore,  a  continuously 
growing  standardized  data  base  of  medical  informa- 
tion, practices,  and  outcomes  will  facilitate  medical 
effectiveness  research. 
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■  Legal,  Confidentiality,  and  Privacy  Issues 


Introduction 

Computer-based  patient  records  bring  into  play 
a  variety  of  laws  that  focus  on  controls  on  the  pri- 
vacy and  confidentiality  of  information  contained  in 
the  records.  The  key  legal  issues  related  to  com- 
puter-based patient  records  include  regulatory  and 
accreditation  issues,  evidentiary  issues,  patient  pri- 
vacy and  record  access  concerns,  record  ownership 
questions,  and  legal  risks  specific  to  computer-based 
patient  record  systems. 

Regulatory  and  Accreditation  Issues 

The  statutes  and  regulations  governing  health 
care  records  are  concerned  with  the  creation, 
authentication,  retention,  storage,  and  media  for 
storage  of  patient  records.  In  addition,  State  regula- 
tions provide  for  confidentiality,  record  content, 
accuracy,  completeness,  timeliness,  and  accessibility 
of  medical  records  (Dick  and  Steen,  1991).  The  anti- 
quated licensing  laws  make  computerized  medical 
records  technically  illegal  in  12  States  and  legally 
ambiguous  in  16  others  (Gardner,  1991b).  The  laws 
are  outdated,  inconsistent,  and  difficult  to  interpret. 
The  wide  variance  in  requirements  for  medical 
records  will  make  it  difficult  to  develop  patient 
record  systems  that  comply  with  laws  in  all  50 
States.  Most  laws  reflect  concerns  about  the  legal 
acceptability  of  the  computerized  medical  record 
as  a  substitute  for  the  handwritten  one  (Roach, 
Chernoff,  and  Esley,  1985).  California  and  Mary- 
land have  comprehensive  medical  record  statutes, 
and  Montana  and  Washington  have  adopted  the 
Uniform  Health  Care  Information  Act. 

States  determine  the  minimum  requirements  for 
the  content  of  medical  records.  The  amount  of  per- 
sonal, financial,  social,  and  medical  data  contained 
in  the  medical  record  varies  by  State.  In  general, 
States  require  the  patient's  consent  to  disclose  the 
contents  of  the  medical  record,  and  the  specifics  are 
State-determined.  Rhode  Island  permits  disclosure, 
with  certain  restrictions,  without  patient  consent 
(Supplement  of  Rhode  Island  General  Laws),  and 
Wisconsin  allows  limited  disclosure  of  information 
in  the  medical  record  without  patient  consent  for 


researchers  affiliated  with  the  provider  (Glenney, 
1989).  In  North  Carolina,  the  patient  may  waive  this 
privilege  (Gellman,  1984). 

Traditionally,  the  physician  s  signature  authenti- 
cated the  medical  record,  and  some  States  still 
require  it.  Other  States  permit  electronic  signatures 
or  the  use  of  a  computer  key  word.  In  Oklahoma, 
physicians  may  use  a  computer  key  to  sign  medical 
records  but  must  use  a  pen  to  order  medications, 
procedures,  or  tests  (Gardner,  1991b).  In  Oregon, 
the  physician  may  sign  orders  with  any  "unique 
identifier"  including  fingerprints.  South  Carolina 
permits  the  use  of  a  signature  stamp  with  "appro- 
priate controls"  (S.C.  Standards  for  Licensing 
Hospitals  and  Institutional  General  Infirmaries 
601.6,  1990);  Iowa  requires  medical  records  to  be 
"written"  and  "signed"  by  the  physician  (Iowa 
Administrative  Code  641.51.5, 1988). 

Laws  governing  the  media  for  storage  and  the 
manner  of  storing  medical  records  exist  in  many 
States  and  relate,  in  part,  to  what  constitutes  admis- 
sible evidence  in  court.  For  example,  in  Texas,  Mis- 
sissippi, Wisconsin,  and  New  Mexico,  medical 
records  must  be  stored  in  the  original  (paper  copy) 
or  on  microfilm  (Gardner,  1991b).  In  some  States, 
the  hard  copy  printout  becomes  the  original  physi- 
cal record  and  is  part  of  the  patient  medical  record. 
Data  stored  on  tapes  must  be  made  visually  intelli- 
gible and  turned  into  written  form  (Tomes,  1990).  In 
all  States,  provisions  must  be  made  for  record  secu- 
rity to  protect  confidentiality  and  to  prevent  alter- 
ation or  falsification  of  the  record.  Few  States  have 
laws  for  altering  medical  records;  those  that  have 
such  laws  require  that  both  the  original  record  and 
the  changes  be  available. 

Personnel  security,  physical  security,  and  system 
security,  including  rules  about  access,  must  be 
assured.  Only  authorized  users  can  have  access  to 
portions  of  the  records  that  are  relevant  to  their 
particular  function.  States  vary  on  who  can  author- 
ize the  use  of  a  medical  record.  Patients,  parents  of 
minors,  Institutional  Review  Boards,  the  State,  and 
others  may  have  the  right  to  authorize  release  of  the 
record.  However,  who  may  authorize  and  when  and 
how  authorization  can  take  place  are  subject  to 
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variability  and  may  be  unclear.  Access  to  sensitive 
records,  such  as  those  relating  to  acquired  immu- 
nodeficiency syndrome  (AIDS)  or  drug  and  alcohol 
abuse  information,  should  be  particularly  well  con- 
trolled. This  is  provided  for  in  the  Privacy  Protec- 
tion Act  for  federally  held  data.  Policies  regarding 
written  records  require  that  medical  records  be  kept 
on  the  premises  of  the  care  provider.  How  this 
affects  computer  storage  of  records  at  remote  sites, 
for  example,  in  shared  facilities  that  reduce  cost  of 
storage,  is  unclear.  It  appears  that  as  long  as  the 
patient  s  privacy  is  protected,  off-site  storage  is  per- 
missible (Gardner,  1991c). 

The  length  of  time  a  record  must  be  kept  varies 
from  5  years  to  permanently,  with  most  States 
requiring  retention  for  5  years  (Roach,  Chernoff, 
and  Esley,  1985).  Record  ownership  also  is  State- 
determined,  with  most  States  agreeing  that  the  facil- 
ity or  provider  owns  the  data,  subject  to  the 
patient's  interest  in  the  information  contained  in  the 
record.  Disclosure  of  medical  records  for  utilization 
review  and  quality  assurance  is  allowed  by  the  State 
and  is  mandated  by  the  Federal  Government  for 
any  federally  funded  providers  (American  Medical 
Record  Association,  1985).  In  Quebec,  joint  owner- 
ship of  the  medical  record  by  the  patient  and  the 
provider  is  provided  for  by  law  (Briere,  1990). 

Use  of  Medical  Records  for  Research 
and  Quality  Assurance 

The  use  of  medical  records  for  research  by  med- 
ical and  nursing  staff  who  are  part  of  the  institution 
collecting  data  and  the  use  of  medical  records  for 
utilization  review  and  quality  assurance  may  depend 
on  the  patient  s  informed  consent,  but  such  use  is 
becoming  more  acceptable  in  most  States,  even 
without  consent  (Gordis  and  Gold,  1980).  A  discus- 
sion of  the  physician's  obligation  to  protect  privacy 
(Gellman,  1984)  noted  that  the  American  Medical 
Association  does  not  address  the  use  of  medical 
records  for  research  and  therefore  the  State,  in  this 
case  North  Carolina,  must  do  so. 

Many  attempts  have  been  made  to  clarify  the 
issues  associated  with  the  use  of  medical  records  for 
research  and  quality  assurance.  Federal  safeguards 
for  the  use  of  medical  information  for  research  are 
reflected  in  the  Privacy  Protection  Study  Commis- 
sion (1977)  report.  Data  held  by  Federal  agencies 
concerning  identifiable  individuals  may  be  released 
without  subject  consent  based  on  interagency  need 


to  know,  for  statistical  records,  and  for  routine  use 
reflecting  a  purpose  compatible  with  purposes  for 
which  the  data  were  collected.  Specific  data  protec- 
tion is  provided  by  a  variety  of  Federal  agencies, 
including  the  Agency  for  Health  Care  Policy  and 
Research  ( AHCPR),  the  Department  of  Veterans 
Affairs,  all  Federal  agencies  that  hold  drug  and 
alcohol  abuse  treatment  data,  Peer  Review  Organi- 
zations, and  others.  An  attempt  to  allow  disclosure 
of  records  without  patient  consent  was  unsuccessful. 
The  American  Medical  Records  Association  (1985) 
prepared  a  position  paper  proposing  the  release  of 
medical  records  for  research  without  patient  con- 
sent under  specified  conditions,  including  review  by 
an  Institutional  Review  Board.  The  American  Sta- 
tistical Association's  Ad  Hoc  Committee  on  Privacy 
and  Confidentiality  (1977)  suggested  that  records  be 
transferred  among  agencies  for  statistical  and 
research  purposes  as  long  as  confidentiality  of  data 
is  protected  and  a  public  purpose  is  served  by  the 
data  transfer. 

Several  organizations  have  been  investigating 
the  legal,  policy,  and  ethical  issues  associated  with 
researcher  access  to  patient  records.  The  Commit- 
tee on  National  Statistics  and  Social  Science 
Research  Council,  Panel  on  Confidentiality  and 
Data  Access,  will  study  the  data  collection  activities 
on  individuals  and  establishments  related  to  sur- 
veys, administrative  record  data,  and  epidemiologi- 
cal studies.  The  Department  of  Health  and  Human 
Services  Task  Force  on  the  Privacy  of  Private  Sector 
Health  Records,  of  which  AHCPR  is  a  part,  is 
examining  the  extent  to  which  there  is  a  problem 
regarding  the  use  of  personally  identifiable  health 
records.  The  American  Public  Health  Association 
Record  Linkage  Work  Group  is  examining  a  variety 
of  public  health  research  and  other  issues  related  to 
data  linkage  (AHCPR,  1991). 

In  Great  Britain,  the  Medical  Research  Council 
(1973,  1985,  as  cited  in  Waller,  1991)  upheld  the  use 
of  medical  records  without  the  patient  s  explicit  con- 
sent as  long  as  confidentiality  was  maintained. 
Canada,  too,  protects  the  confidentiality  of  the 
patient  while  allowing  release  of  medical  records  for 
research  with  the  patient's  consent. 

In  summary,  there  is  general  acceptance  of  the 
use  of  medical  records  for  research  and  quality 
assurance.  However,  there  does  not  appear  to  be 
agreement  on  whether  the  patient  must  consent  to 
its  use.  In  all  cases,  it  is  assumed  that  patient  confi- 
dentiality will  be  maintained  and  that  reasons  for 
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use  are  justified.  This  allows  for  wide  latitude  in  the 
use  of  medical  record  information. 

Inherent  in  the  ability  to  use  medical  records  for 
research  or  quality  assurance  is  the  need  for  record 
linkage.  The  American  Public  Health  Association  s 
Record  Linkage  Work  Group  is  examining  the 
issues  related  to  linkage  including:  confidentiality 
protection  in  Federal.  State,  and  private  sector  are- 
nas; the  effect  of  confidentiality  on  provider  report- 
ing; ownership  of  linked  files  and  their  subsequent 
use;  and  the  use  of  individual  identifiers,  scrambled 
identifiers,  or  data  stripped  of  identifiers. 

Patients9  rights.  The  Federal  Privacy  Act 
(Health  Care  Financing  Administration,  1974),  sup- 
ported by  similar  acts  in  many  States,  provided 
assurance  that  patient  records  held  by  the  Federal 
and  State  Governments  will  not  be  disclosed  to 
third  parties  without  the  patient's  consent,  except 
under  specific  conditions  (U.S.C.  552a).  A  wide 
variety  of  statutes  govern  patients'  rights,  and  many 
State  laws  neither  cover  nor  exclude  records  held  in 
the  private  sector.  There  is  an  increasing  demand 
for  information  about  the  social,  financial,  demo- 
graphic, and  medical  data  contained  in  the  medical 
record.  In  addition  to  peer  review  bodies,  third- 
party  payers,  outside  billing  and  computer  services, 
employers,  insurers,  and  others  who  use  health  care 
information  for  non-health-care  purposes  want 
access  to  the  medical  record  (Waller,  1991). 

Computerization  of  medical  records  makes  it 
difficult  to  control  the  redisclosure  of  data  to  other 
parties  and  the  abuses  inherent  in  the  uncontrolled 
use  of  that  information.  Information  that  crosses 
State  lines  will  not  be  subject  to  the  same  con- 
straints held  within  the  originating  State.  As 
patients  and  providers  become  more  aware  of  the 
potential  for  misuse  of  medical  record  information, 
the  medical  record  itself  may  be  compromised.  Pro- 
fessionals and  patients  may  be  unwilling  to  provide 
complete  or  accurate  information  about  sensitive 
issues  such  as  AIDS,  psychiatric  problems,  or  alco- 
hol and  drug  abuse  (Burnum,  1989). 

Privacy  and  confidentiality  issues  are  of  great 
concern  in  anticipating  the  use  of  automated  med- 
ical records.  Computers  increase  the  amount  of 
information  collected  (Gellman,  1984),  the  number 
of  people  who  have  access  to  the  data,  and  the 
degree  to  which  the  data  are  shared  and  exchanged 
(Andreoli  and  Musser,  1985;  Goldman,  1987).  Pri- 
vacy is  defined  as  the  right  of  an  individual  to  con- 
trol the  amount  of  information  he  divulges  about 


himself.  Confidentiality  describes  how  the  informa- 
tion, once  collected,  will  be  treated  (Trute  and 
Tonn,  1982).  To  protect  privacy  and  confidentiality, 
it  is  necessary  to: 

1.  Separate  data  sufficiently  from  individual  identi- 
fiers and  to  aggregate  data  sufficiently  when 
reporting  on  them  to  avoid  inadvertent  disclo- 
sure of  information. 

2.  Limit  the  collection  of  data  to  necessary  items 
only. 

3.  Provide  for  security  of  the  computer  systems. 

4.  Permit  access  to  data  only  on  a  "need  to  know" 
basis. 

Although  each  State  supports  the  individuals 
right  to  privacy  and  confidentiality,  the  individual  is 
not  well  protected.  The  Uniform  Health  Care  Infor- 
mation Act  (National  Conference  of  Commission- 
ers on  Uniform  State  Laws,  1985)  detailed  the  man- 
ner in  which  patient  information  may  be  released. 
Only  Montana  (M.C.A.  50-16-501)  and  Washington 
(as  of  July  1991)  have  enacted  this  into  law 
(Andreoli  and  Musser,  1985).  Providers  are  oblig- 
ated by  State  licensure  laws  and  statutes  to  maintain 
the  confidentiality  of  patient  records  and  protect 
them  from  unauthorized  release  whether  they  are 
maintained  on  paper  or  computer.  As  described  ear- 
lier, however,  it  is  much  more  difficult  to  protect  and 
control  access  to  computer  records. 

Patients  always  have  access  to  portions  of  the 
hospital  record  that  contain  their  data,  but  their 
right  of  access  to  records  held  by  physicians  and 
other  individual  providers  is  not  as  clear.  Access  is 
generally  granted  in  response  to  a  written  request; 
however,  only  a  summary  of  the  medical  record  may 
be  made  available.  Again,  who  grants  authorization 
and  under  what  conditions  and  to  whom  it  is 
granted  is  variable  and  often  unclear.  Providers  may 
prevent  patient  access  to  psychiatric  records  but 
provide  those  records  to  an  appointed  representa- 
tive. Patients  may  not  even  know  about  the  exis- 
tence of  insurers"  and  others'  data  bases  and  there- 
fore not  be  able  to  enforce  their  right  to  access.  In 
addition,  once  access  is  gained,  the  patient  may  not 
have  the  right  to  correct  inaccuracies  in  the  record. 
These  restrictions  on  the  patient  s  right  to  access 
may  limit  his  or  her  willingness  to  support  an  auto- 
mated medical  record. 
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Providers'  rights.  The  provider,  whether  institu- 
tional or  individual,  may  have  concerns  about  the 
privacy  and  confidentiality  of  the  medical  record — 
for  example,  they  may  be  concerned  that  increasing 
access  by  patients  and  others  to  the  medical  record 
will  lead  to  increased  litigation  related  to  practice 
patterns  and  patient  outcomes.  They  are  likely  to  be 
concerned,  too,  about  the  security  of  the  records 
and  the  chance  for  falsification  or  abuse  of  their 
contents.  Improper  aggregation  of  data  and/or  fail- 
ure to  separate  identifying  information  from  other 
data  could  lead  to  inadvertent  disclosure  of  the 
provider's  identity. 

Security  policies  and  confidentiality. 

Security  of  automated  medical  records  has  been 
addressed  by  many  of  those  who  have  implemented 
automated  ambulatory  medical  records  systems. 
These  security  measures  focus  on  passwords, 
authorized  access,  and  levels  of  access.  For  exam- 
ple, someone  may  have  read-only  access  to  the 
data,  or  there  may  be  usage  monitoring  and  warn- 
ings regarding  unauthorized  use. 

In  the  University  of  Pittsburgh  (Presbyterian 
Hospital)  system,  initial  policies  that  required 
dial-back  security  were  unworkable.  The  System 
Security  Study  Committee  suggested  several  steps 
for  computer  system  security  (National  Research 
Council,  1991): 

•  Promulgation  of  a  comprehensive  set  of 
Generally  Accepted  System  Security  Principles 
(GASSP). 

•  A  set  of  short-term  actions  for  system  vendors 
and  users  that  builds  on  readily  available  capa- 
bilities and  would  yield  immediate  benefits. 

•  A  system-incident  data  repository  and  appropri- 
ate education  and  training  programs  to  promote 
public  awareness. 

•  Clarification  of  export  control  criteria  and  pro- 
cedures for  secure  or  trusted  systems  and  review 
for  possible  relaxation  of  controls  on  the  export 
of  implementations  of  the  Data  Encryption 
Standard  (DES). 

•  Funding  and  directions  for  a  comprehensive 
program  of  research. 

•  A  new  organization  to  nurture  the  development, 
commercialization,  and  proper  use  of  trust 
technology,  referred  to  as  the  Information 
Security  Foundation,  or  ISF. 


Evidentiary  Issues 

Computerized  medical  records  can  be  struc- 
tured to  be  admissible  as  evidence  in  court  in  dis- 
putes between  providers  and  patients  or  payers. 
Computer-based  records  are  considered  to  be 
hearsay  evidence  unless  one  of  the  many  excep- 
tions apply.  For  example,  if  computer  records  are 
kept  regularly  in  the  ordinary  course  of  care,  they 
can  be  admitted  as  evidence  in  court.  In  addition, 
the  original  record  must  be  maintained,  each  per- 
son who  makes  an  entry  or  modifies  a  record  must 
be  identified,  and  the  changes  to  the  record  also 
must  be  identified  (Waller,  1991). 

Liability  Issues 

Computer  programs  are  considered  to  be  prod- 
ucts because  they  are  owned,  exist  through  time, 
may  have  errors  that  can  be  corrected,  and  can  be 
passed  from  one  person  to  another.  They  are  there- 
fore subject  to  strict  liability.  The  computer  can  pro- 
duce information  which  the  provider  accepts  without 
verification,  such  as  laboratory  results,  and  the  pro- 
gram can  be  liable  for  errors.  A  defect  in  a  computer 
system  that  results  in  logic  errors  or  programming 
errors  can  cause  liability.  Systems  most  likely  to  be 
involved  in  litigation  include  patient  record  and 
treatment  support  systems  because  they  contain 
many  different  types  of  information,  operate  in  real 
time,  and  utilize  remote  terminals  (Brannigan,  1989). 

Summary 

Confidence  in  the  ability  to  protect  privacy  and 
confidentiality  will  reduce  the  fear  of  computers 
for  patients  and  providers  (Bank  and  Laska,  1978). 
Patient  rights,  provider  rights,  and  evidentiary  and 
liability  concerns  will  be  addressed  by  ensuring 
that  inadvertent  disclosure  of  the  contents  of  the 
medical  record  will  not  occur  and  that  the  contents 
wili  not  be  altered.  Uniform  national  standards  for 
computer-based  records  and  record  systems  need 
to  define  creation,  authentication,  storage,  and 
retention  of  patient  records.  Adoption  of  a  Uni- 
form Health  Care  Information  Act  would  provide 
patients  access  to  their  health  records,  the  ability 
to  correct  information,  and  the  guarantee  of  confi- 
dentiality across  State  lines.  It  would  define  the 
rights  and  obligations  of  third  parties  who  receive 
the  information  and  prevent  disclosure  or  abuse  of 
patient  information.  These  issues  will  need  to  be 
considered  in  the  development  of  an  automated 
ambulatory  medical  records  data  base  to  support 
medical  effectiveness  research. 
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■  Methodological  Issues 


Introduction 

Available  data  bases  may  contain  useful  data  on 
ambulatory  care:  insurance  claims  data  bases,  ran- 
domized clinical  trial  (RCT)  data  bases,  intramural 
research  data  bases,  administrative  or  health  care 
management  data  bases,  clinical  data  bases,  and 
quality  review  data  bases.  Despite  their  differences, 
these  data  bases  can  be  used  to  generalize  from  the 
individual  to  the  patient  population.  The  utility  of 
each  is  based  on  the  degree  to  which  this  generaliz- 
ability  is  possible.  The  data  base  s  structure,  accessi- 
bility, quality,  and  the  manner  in  which  the  data  are 
stored  all  contribute  to  its  use.  Combining  data  from 
data  bases  to  address  issues  of  interest  requires  con- 
sistent definition  of  data  items  and  quality  data 
(Muhlbaier  and  Pryor,  1989). 

Of  the  potential  sources  of  data  on  ambulatory 
care  for  the  study  of  medical  effectiveness,  two 
have  emerged  from  existing  research  studies — clini- 
cal trials  and  clinical  data  bases  derived  from  obser- 
vational studies.  Clinical  trials  are  driven  by 
research  protocols;  ambulatory  care  data  collected 
during  the  course  of  a  clinical  trial  may,  therefore, 
be  suitable  for  medical  effectiveness  studies.  Clini- 
cal data  bases  contain  observational  data  collected 
as  a  routine  part  of  patient  care;  patients  are  not 
randomized  and  intervention  is  not  prescribed 
(Kunitz,  Gross,  and  Heyman,  1981).  However,  sev- 
eral of  the  existing  clinical  data  bases  are  the  prod- 
ucts of  observational  studies  that  have  utilized  sys- 
tematic data  collection  procedures,  and  they  have 
been  used  to  support  a  wide  variety  of  research 
endeavors.  Examples  include  the  Duke  Cardiovas- 
cular Diseases  Data  Bases  and  the  ARAMIS 
(Arthritis,  Rheumatism,  and  Aging  Medical  Infor- 
mation System)  data  bases.  These  clinical  data 
bases  have  been  used  for  epidemiological  studies, 
such  as  the  assessment  of  patient  risk  and  postmar- 
keting surveillance  to  evaluate  adverse  effects  when 
a  drug  is  used  in  general  practice.  For  example,  the 
ARAMIS  data  base  was  used  to  study  the  effects  of 
azathioprine  in  patients  with  rheumatoid  arthritis 
(Tierney  and  McDonald,  1991).  In  this  application, 
instead  of  relying  on  the  physician  to  report 
adverse  effects  from  the  use  of  this  new  drug, 


patients  given  early  prescriptions  of  the  drug  noted 
in  the  data  base  were  surveyed  to  get  a  more  com- 
plete picture  of  the  responses  (Moses,  1991 ). 

Practice  data  bases  are  another  form  of  clinical 
data  base  and  are  distinguished  from  observational 
data  bases.  The  practice  data  base  contains  data  col- 
lected as  a  part  of  routine  care  but  not  necessarily 
according  to  a  common  data  collection  protocol. 
These  data  bases  reflect  variations  in  physician 
practice,  resource  consumption,  quality  assurance, 
and  the  effects  of  cost  reduction  on  quality  of  care. 
In  a  study  at  the  San  Francisco  Veterans  Adminis- 
tration Hospital,  the  practice  data  base  was  a  better 
predictor  of  hospital  costs  than  were  the  diagnosis- 
related  groups  (DRGs)  (Tierney  and  McDonald, 
1991).  Practice  data  bases  have  been  used  in  physi- 
cian decisionmaking  and  in  collaborative  clinical 
research.  In  the  latter,  they  have  served  as  a  pool  of 
subjects  for  clinical  research  protocols. 

Advocates  of  using  clinical  trial  data  (Byar, 
1991)  find  a  number  of  problems  with  observational 
and  practice  data  bases  that,  in  their  opinion,  com- 
promise their  utility.  Although  it  is  true  that  obser- 
vational data  bases  are  compiled  by  health  care 
institutions,  contain  timely  data  generated  during 
the  normal  course  of  care,  and  pool  data  from  a 
variety  of  institutional  sources  such  as  the  labora- 
tory, pharmacy,  and  medical  record,  they  lack  the 
randomization  and  treatment  intervention  neces- 
sary for  determining  treatment  efficacy.  They  are, 
however,  useful  for  medical  effectiveness  research. 

Three  major  issues  need  to  be  addressed  in  the 
use  of  practice  data  bases:  data  issues,  patient 
issues,  and  bias. 

Data  Issues 

In  general,  practice  data  bases  were  not  devel- 
oped for  the  research  purposes  for  which  they  are 
used.  Research  questions  are  not  generally  formu- 
lated prior  to  data  collection,  and  therefore  data 
may  not  be  available  to  answer  these  questions 
(Pryor  and  Lee,  1991).  Data  are  recorded  only  when 
patients  contact  a  particular  aspect  of  the  health 
care  system.  Therefore,  data  may  be  missing 
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because  (1)  providers  are  outside  of  that  particular 
part  of  the  system,  (2)  the  patient  chooses  not  to 
visit  a  health  care  facility,  or  (3)  eligibility  criteria 
have  changed,  and  the  patient  can  no  longer  receive 
health  care  there.  It  is  therefore  not  possible  to 
judge  if  the  patient  is  not  visiting  that  particular 
health  care  provider,  is  being  treated  elsewhere,  or 
is  not  being  treated  at  all.  In  addition,  patients  who 
are  less  ill  may  visit  the  health  care  facility  or 
provider  less  often,  and  thus,  data  may  be  more  rep- 
resentative of  sicker  patients.  Patients  and  therapies 
are  not  randomly  assigned,  and  the  resulting  bias 
can  seriously  affect  the  quality  of  the  data  (Tierney 
and  McDonald,  1991). 

The  contents  of  the  data  sets  from  different  sites 
may  vary  considerably.  Consistent  definition  of  vari- 
ables and  quality  control  are  minimum  require- 
ments for  aggregating  data  (Muhlbaier  and  Pryor, 
1989).  The  lack  of  consistency  requires  that  atten- 
tion be  given  to  quantifying  and  controlling  for  dif- 
ferences in  the  data.  It  is  likely  that  the  schedule  for 
data  collection  will  also  vary  across  sites,  and  thus, 
interpretation  of  the  effect  of  interval  differences  in 
data  collection  must  be  assessed. 

Decisions  as  to  the  comparable  value  of  multi- 
ple measures  of  the  same  parameter  must  be  made, 
and  a  plan  to  operationalize  that  choice  must  be 
implemented  carefully  (Tierney  and  McDonald, 
1991).  In  addition,  a  decision  as  to  the  level  at 
which  measures  are  aggregated  is  important 
(McDonald  and  Hui,  1991). 

Missing  data  present  a  particular  problem  in  the 
use  of  observational  data  bases.  Missing  data  may 
reflect  data  not  collected,  data  lost  or  garbled  at 
entry  or  transmission,  or  data  that  are  not  applica- 
ble for  a  variety  of  reasons  (McDonald  and  Hui, 
1991).  The  mechanism  for  selection  of  missing  data 
is  generally  not  known,  and  bias  can  be  introduced 
into  the  study. 

Patients'  Issues 

Historically,  practice  data  bases  have  contained 
fewer  patient. data  than  have  claims  data  bases,  and 
they  have  been  limited  to  the  individual  facility 
housing  the  system.  Larger  data  bases,  like  claims 
data  bases,  are  more  sensitive  to  low  frequency 
medical  conditions,  but  practice  data  bases  are  more 
likely  to  identify  clinical  changes  in  patients.  Most 
practice  data  bases,  especially  research-focused 
ones,  are  located  in  academic  medical  centers  and 


contain  information  on  patients  who  are  not  repre- 
sentative of  the  general  population;  that  is,  they  are 
more  likely  to  have  a  lower  socioeconomic  status. 
Providers  are  more  likely  to  be  residents  and  their 
teachers,  making  them  unrepresentative  of  most 
practicing  physicians.  However,  these  limitations  of 
practice  data  bases  will  be  addressed  when  collabo- 
ration between  academia  and  general  practice 
occurs  with  more  regularity,  and  individual  data 
bases  are  linked  to  provide  a  much  larger  sample 
(Tierney  and  McDonald,  1991). 

Bias 

Selection  of  patients  by  providers  for  diagnostic 
procedures  and  treatments  introduces  bias  into  the 
natural  history  of  disease  and  treatment  (Tierney 
and  McDonald,  1991).  Obviously,  physicians  will 
choose  only  patients  for  whom  they  think  a  treat- 
ment will  be  beneficial  (McDonald  and  Hui,  1991). 
This  is  one  of  the  shortcomings  of  observational 
studies  (Byar,  1991).  Models  can  be  derived  to  con- 
trol for  this  bias  and  negate  the  deficiency  in  obser- 
vational studies. 

Although  RCTs  decrease  the  treatment  bias  by 
using  random  allocation  of  patients,  bias  may  exist 
in  RCTs  as  well  as  in  observational  studies.  Partici- 
pation in  an  RCT  is  based  on  well-defined  eligibility 
criteria  that  define  a  uniform  population  for  study. 
This  may  limit  the  generalizability  of  the  results  of 
the  study.  In  the  Coronary  Artery  Surgery  Study 
(CASS)  (Kronmal,  Davis,  Fisher,  and  others,  1978), 
patients  who  were  entered  into  the  RCT  were  com- 
pared with  those  who  met  the  eligibility  criteria  but 
were  not  randomized.  Researchers  found  that  ran- 
domized patients  differed  from  randomizable 
patients  in  prognostic  factors,  especially  in  severity 
of  cardiovascular  disease  and  symptoms. 

Until  recently,  randomization  was  done  in  an 
unblocked  fashion  which  often  produced  subsets 
that  were  not  balanced  in  all  strata.  Nonsignificant 
imbalances  in  powerful  prognostic  factors  can  lead 
to  bias.  Now  that  randomization  is  performed  in 
blocks  within  the  identified  subsets,  the  introduction 
of  bias  in  this  manner  is  less  likely  to  occur. 

If  subgrouping  of  patients  in  an  RCT  is  per- 
formed after  randomization  has  taken  place,  results 
may  be  misinterpreted.  This  is  especially  true  when 
overall  group  differences  are  not  significant,  or  only 
marginally  significant,  and  the  subsets  are  small. 
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A  more  technical  cause  of  bias  lies  in  the  use  of 
nonlinear  statistical  models  in  RCTs.  These  models 
may  yield  an  estimate  of  treatment  effect  when 
important  covariates  are  accidentally  omitted 
(because  they  are  not  identified)  from  statistical 
analyses.  In  addition,  failure  to  recognize  if  there  is 
a  treatment  by  covariate  interaction  in  both  linear 
and  nonlinear  models  may  yield  bias  (Hlatky,  Lee, 
Harrell,  and  others,  1984). 

Addressing  Problems  with  Practice 
Data  Bases 

A  number  of  methodological  issues  relate  to  the 
use  of  automated  data  bases  in  ambulatory  care 
effectiveness  studies.  Research  using  data  bases 
would  be  enhanced  if  methods  for  acquiring  better 
data  were  designed.  Prospectively  collected  data, 
with  clear  definition  and  detailed  specifications, 
and  a  protocol  for  their  collection  would  be  valu- 
able. Careful  choice  of  collaborators  and  sites  and  a 
plan  to  control  or  allow  for  interfering  variables 
would  improve  data  quality  (Moses,  1991).  This 
description  conforms,  in  many  ways,  with  that  of  a 
clinical  trial.  In  the  observational  data  base,  how- 
ever, all  the  patients'  data  would  be  included,  and 
no  attempt  to  select  patients  or  randomize  them  to 
treatments  would  occur. 

Time,  as  a  formal  dimension,  is  an  important 
aspect  of  longitudinal,  observational  data  bases  in 
which  time  series  analysis  across  many  patients  is 
necessary.  Analyses  of  trends  over  time  for  mortal- 
ity, morbidity,  disability,  expenditures  for  health 
care,  and  variations  over  geographic  areas  have 
been  carried  out.  Krakauer  and  Bailey  (1991)  have 
performed  univariate  time  to  event  (treatment, 
hospitalization,  readmission,  death)  on  Health  Care 
Financing  Administration  files  (Medicare  and  Med- 
icaid) to  monitor  trends  over  time  and  assess  inter- 
ventions. 

A  complete  uniform  data  set  permits  the  most 
credible  and  productive  analysis,  and  therefore, 
attempts  to  deal  with  missing  data  are  of  particular 
importance.  As  described  earlier,  a  prospective 
study  with  standardized  definitions  would  assist  in 
the  collection  of  a  complete  set  of  data.  In  some 
cases,  prospective  data  collection  was  undertaken  to 
augment  data  from  a  practice  data  base  to  deal  with 
missing  data;  however,  this  is  an  expensive  and  not 
always  practical  approach.  In  other  cases,  objective 
data  are  substituted  for  subjective  data.  For  exam- 


ple, a  chest  x-ray  can  reflect  a  diagnosis  of  conges- 
tive heart  failure  even  when  the  diagnosis  is  missing 
from  the  medical  record.  A  study  assessing  the  qual- 
ity of  data  excerpted  from  medical  records  found 
that  subjective,  ambiguous,  or  incomplete  data  were 
subject  to  the  most  errors  in  excerpting  and  coding, 
resulting  in  a  great  deal  of  missing  or  unusable  data 
(Horowitz  and  Yu,  1984). 

Missing  data  have  sometimes  been  addressed  by 
supplementing  the  practice  data  base  with  data 
from  a  claims  and/or  national  data  base.  This  link 
to  a  large  data  base  allows  for  inference  of  missing 
data.  It  does,  however,  assume  that  definitions 
across  data  bases  are  standardized.  The  Agency  for 
Health  Care  Policy  and  Research  and  the  Ameri- 
can Society  for  Testing  and  Materials,  a  national 
consensus  standards  organization,  are  attempting  to 
develop  a  standard  structure  and  format  for  trans- 
mitting clinical  data  (McDonald  and  Hui,  1991).  In 
some  studies,  variables  and/or  patients  with  missing 
data  have  been  eliminated  from  the  study.  How- 
ever, this  may  introduce  bias  or  compromise  the 
power  of  analysis  and  therefore  is  not  an  approach 
of  choice  (Pryor  and  Lee,  1991). 

In  many  studies,  missing  data  are  addressed  by 
imputing  a  value  from  the  respondent's  mean  or 
from  a  predicted  value.  Safran  (1991)  suggested  that 
multiple  imputation  for  nonresponses  is  a  more  reli- 
able method  for  dealing  with  missing  data.  In  this 
way,  missing  values  can  be  replaced  by  two  or  more 
plausible  values.  The  values  can  represent  uncer- 
tainty about  which  values  to  impute,  assuming  the 
reasons  for  nonresponse  are  known. 

To  enhance  the  predictive  method,  assess  the 
quality  of  the  prediction,  and  design  a  validation 
strategy,  it  may  be  necessary  to  employ  innovative 
methods  of  analysis.  Given  the  different  types  of 
data  available  (for  example,  binaiy,  ordinal,  contin- 
uous), it  is  possible  to  use  such  methods  as  sub- 
grouping  or  stratification,  regression  modeling 
strategies  to  adjust  for  large  numbers  of  variables, 
Bayesian  approaches,  pattern  recognition,  and 
dynamic  approaches. 

Innovative  methods  for  improving  predictive 
methods  include:  employing  less  restrictive  methods 
regarding  the  distribution  of  dependent  and  inde- 
pendent variables  and  their  relationship  to  each 
other,  using  superior  information  for  dependent  and 
independent  variables,  adjusting  for  differences 
between  groups,  using  dynamic  approaches  such  as 
repeated  measures  and  time  covariates,  using  more 
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clinically  interpretable  data,  and  predicting  the 
likely  yield  of  tests  before  they  are  ordered.  In  addi- 
tion, innovations  within  methods  are  suggested  to 
verify  the  model  assumptions  about  the  distribution 
of  variables  involved  or  the  nature  of  the  relation- 
ship between  predictor  variables  and  outcome  vari- 
ables (Pryor  and  Lee,  1991). 

A  method  is  needed  to  assess  and  compare  the 
quality  of  predictors.  For  this,  it  is  necessary  to  com- 
pare precision  or  repeatability,  including  the  confi- 
dence limits  of  a  prediction,  the  reliability  (that  is, 
how  close  a  prediction  is  to  the  actual  value  for  a 
patient),  and  discrimination  (that  is,  the  ability  to 
separate  patients  with  or  without  a  specific  outcome). 

Validation  strategies  should  address  the  general- 
izability  of  a  prediction.  Internal  methods  of  gener- 
alization provide  for  use  of  the  model  on  the  same 
population  for  whom  it  was  developed  to  validate 
the  model.  With  this  method,  applicability  of  find- 
ings to  new  patients  cannot  be  described.  External 
methods  of  validation  apply  the  model  to  new  and 
different  populations  to  evaluate  its  performance 
(Pryor  and  Lee,  1991). 
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6  ■  Selected  Automated  Ambulatory  Medical  Records  Systems 


This  chapter  describes  selected  automated 
ambulatory  medical  records  systems  ( AAMRSs) 
and  related  systems  that  have  potential  to  support 
medical  effectiveness  research  in  ambulatory  set- 
tings. The  descriptions  focus  on  the  system  of 
records  or  the  computer  and  manual  processes  that 
surround  automated  recordkeeping  including 
inputs,  outputs,  and  processes. 

Included  in  the  descriptions  are  the  following 
systems:  ARAMIS,  ATHOS,  BICS,  BIHS,  CIS, 
COSTAR,  DIOGENE.  HCHP/AMRS,  HELP, 
MARS,  PIMS/PIPS,  RMRS,  RPMS,  STOR, 
THERESA,  TMR,  and  VIPOR. 

Descriptions  are  based  on  a  literature  review,  as 
well  as  information  from  the  system  "owners"  and 
not  on  an  indepth  study  of  the  documentation  of 
each  system.  As  indicated  in  the  text  of  each 
description,  most  of  these  systems  have  not  been 
applied  to  medical  effectiveness  studies,  perhaps 
because  this  is  an  emerging  field.  The  systems,  how- 
ever, do  contain  clinical  data,  documenting  the  clini- 
cal course  of  a  patient,  and  could  be  useful  for  med- 
ical effectiveness  studies.  The  utility  of  these 
systems  will  generally  be  enhanced  if  medical 
records  systems  "owners"  and  clinicians  agree  to 
administer  outcomes  questionnaires  to  their 
patients. 

System:  ARAMIS 

Arthritis,  Rheumatism,  and  Aging  Medical 

Information  System 
Stanford  University  Medical  Center,  Palo  Alto,  CA 
Primary  Contact:  J.  Fries,  M.D. 

This  system  was  developed  to  support  clinical 
research  in  chronic  diseases  and  is  a  prototype  of  a 
national  chronic  disease  data  bank.  ARAMIS  con- 
tains longitudinal,  observational  data  for  rheumatic 
disease  obtained  from  multiple,  collaborating,  uni- 
versity-based and  private  medical  clinic  settings  col- 
lected with  the  Time-Oriented  Record  (TOR).  It 
provides  a  uniform  data  base  of  rheumatic  diseases. 
Data  reflect  stratifying,  intervention,  and  outcome 
measures.  The  system  has  a  descriptive  vocabulary 


and  data  dictionary  with  a  series  of  associated  glos- 
saries and  administrative  manuals  documenting 
ascertainment  and  scoring  rules  for  all  data  elements 
within  the  data  base.  As  of  1991,  ARAMIS  con- 
tained 17  data  banks  from  the  United  States  and 
Canada.  Within  the  system,  data  are  shared  on  over 
25,000  patients. 

Characteristics.  The  system  was  developed  from 
1970  to  1975.  Currently,  the  data  banks  reside  in 
Medlog,  a  commercial  product  developed  by  Infor- 
mation Analysis  Corporation  from  the  original  data 
base  management  system  TOD  (Time-Oriented 
Database).  Medlog,  written  in  C,  supports  both 
direct  and  remote  data  entry,  editing,  retrieval, 
statistical  analysis,  and  graphics. 

Data  include  laboratory  and  therapeutic  infor- 
mation as  well  as  other  quantitative  data  and  quali- 
tative clinical  observations.  Medical  histories  are 
self-administered  by  patients  and  are  reviewed  by 
physicians.  In  addition,  a  Health  Assessment  Ques- 
tionnaire (HAQ)  is  used  on  a  regular  basis  to  collect 
information  on  disability,  discomfort,  iatrogenic 
conditions,  economic  impact,  and  mortality.  Data 
are  entered  from  TORs,  used  by  clinicians  in  each 
of  the  clinical  entities,  or  ported  from  computer- 
based  medical  records  systems. 

Retrieval  is  accomplished  through  a  transposed 
file  using  time-oriented  subsetting  and  analytic 
techniques.  Data  may  be  ported  to  the  standard 
statistical  packages  SAS,  BMPD,  or  SPSS  or  may  be 
analyzed  directly  via  a  series  of  built-in  interactive 
statistical  programs  within  Medlog. 

Comments.  Although  ARAMIS  is  limited  to  arthri- 
tis and  rheumatology,  the  system  contains  data  from 
multiple  ambulatory  care  settings  and  provides  a 
model  for  a  chronic  disease  ambulatory  care  and 
research  data  base.  Studies  conducted  on  clinical 
research  have  produced  over  477  publications. 
Although  ARAMIS  is  not  an  automated  medical 
records  system,  since  it  was  not  developed  to  sup- 
port care,  therapeutic  effects  have  been  studied  and 
the  data  contain  many  of  the  same  features  as  those 
collected  through  AAMRSs. 
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System:  ATHOS 

AIDS  Time-Oriented  Health  Outcome  Study 
Stanford  University  Medical  Center 
Palo  Alto,  CA 

Primary  Contact:  J.  Fries,  M.D. 

This  system  was  developed  under  a  grant  from 
the  Agency  for  Health  Care  Policy  and  Research 
(AHCPR)  to  support  clinical  research  and  health 
services  research  in  acquired  immunodeficiency  syn- 
drome (AIDS)  and  human  immunodeficiency  virus 
(HIV)  infection  and  is  a  prototype  of  a  national 
long-term  AIDS  data  bank.  It  contains  longitudinal, 
observational  data  for  HIV  and  resulting  disorders 
obtained  from  multiple  collaborating  private  prac- 
tice and  university-based  clinics.  Data  are  collected 
via  TORs  or  from  electronic  medical  records  sys- 
tems (MAC-based  Therapy  Management  System 
[TMS]  or  DOS-based  QD  Patient  Management  Sys- 
tem). Data  reflect  stratifying,  intervention,  and  out- 
come measures  in  AIDS.  It  has  a  descriptive  vocab- 
ulary and  data  dictionary  with  a  series  of  associated 
glossaries  and  administrative  manuals  documenting 
ascertainment  and  scoring  rules  for  all  data  ele- 
ments within  the  data  base.  As  of  1991,  ATHOS 
contained  five  data  banks  from  San  Francisco,  Los 
Angeles,  and  San  Diego.  Within  the  system,  data 
are  shared  on  over  1,500  patients. 

Characteristics.  The  system  was  developed  in  1989. 
Currently,  the  data  banks  reside  in  Medlog,  a  com- 
mercial product  developed  by  Information  Analysis 
Corporation  from  the  original  data  base  manage- 
ment system.  Medlog,  written  in  C,  supports  both 
direct  and  remote  data  entry,  editing,  retrieval,  sta- 
tistical analysis,  and  graphics. 

Data  include  laboratory  and  therapeutic  infor- 
mation, as  well  as  other  quantitative  data  and  quali- 
tative clinical  observations.  Medical  histories  and 
physical  findings  are  obtained  by  physicians  in  the 
clinical  care  setting  using  paper  forms  or  direct  input 
into  the  computerized  patient  management  system. 
In  addition,  a  baseline  health  history  and  self-admin- 
istered, quarterly  health  assessment  questionnaires 
(HAQs)  are  obtained  from  patients  to  collect  infor- 
mation on  disability,  pain,  quality  of  life,  side  effects, 
economic  impact,  health  care  utilization,  and  mortal- 
ity. Data  are  entered  from  the  questionnaires  by 
clinic-based  outcome  assessors  or  directly  by  physi- 
cians into  the  patient  management  system  with  sub- 
sequent porting  to  the  Medlog  data  banks. 


Retrieval  is  accomplished  through  a  transposed 
file  using  time-oriented  subsetting  and  analytic 
techniques.  Data  may  be  ported  to  the  standard  sta- 
tistical packages  SAS  or  BMDP  or  analyzed  directly 
via  a  series  of  built-in  interactive  statistical  programs 
within  Medlog. 

Comments.  Although  ATHOS  is  limited  to  AIDS- 
and  HIV-related  disorders,  the  system  contains  data 
from  multiple  outpatient,  inpatient,  and  ambulatory 
care  settings  and  provides  a  model  for  observational 
data  bases  in  AIDS.  Studies  are  ongoing  and  in- 
clude 10  publications  and  conference  presentations 
to  date.  Use  of  the  TMS  and  QD  systems  as  part  of 
this  project  provides  prototypes  for  automated  elec- 
tronic medical  records  systems  used  to  support 
patient  care.  The  overall  approach  to  health  services 
research  follows  that  of  ARAMIS  in  terms  of  view- 
ing long-term  outcomes. 

System:  BICS 

Brigham  and  Women's  Hospital 

Integrated  Computing  System 
Boston,  MA 

Primary  Contact:  J.  Teich,  M.D. 

The  Brigham  and  Women's  Hospital  Integrated 
Computing  System  (BICS)  contains  broad  clinical 
and  administrative  data  bases  which  are  used  by 
nearly  all  departments  in  the  hospital.  The  current 
version  of  BICS  stems  from  the  Beth  Israel  Hospi- 
tal system,  which  was  installed  at  Brigham  and 
Women's  Hospital  (BWH)  in  1985;  BICS  has  been 
developed  independently  at  BWH  since  1988.  The 
ambulatory  record  was  developed  in  1989  and  now 
serves  many  of  the  hospital's  practices  with  stations 
in  every  examination  room.  A  new  satellite  ambu- 
latory center,  located  a  few  miles  from  the  hospital, 
is  also  served  by  the  computerized  record  system 
through  fiberoptic  links.  Specific  ambulatory  record 
information  is  currently  available  on  22,000 
patients;  other  clinical  information,  organized  and 
presented  for  ambulatory  use,  is  available  on  any 
hospital  patient. 

Characteristics.  BICS  runs  on  a  network  of  Data 
General  Eclipse  Computers,  programmed  in  the 
MIIS  dialect  of  MUMPS.  Conversion  of  the  entire 
system  to  a  network  of  microcomputers,  running 
standard  MUMPS,  is  underway  and  will  continue 
in  1993. 

Data  content  includes  condition,  complications, 
medications,  notes,  allergies,  health-maintenance 
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information,  vital  signs,  and  visit  summaries.  The 
visit  summaries  organize  all  available  clinical  data  in 
a  visit-by- visit  fashion  for  review.  Information  can 
also  be  reviewed  by  problem  and  by  data  type.  A 
provider's  to-do  list  is  also  available  for  each  patient. 
Specific  flowsheets  can  be  constructed  for  use  in  a 
given  practice  to  handle  specific  data  elements  and 
viewpoints  needed  by  that  particular  practice. 

Data  are  entered  by  (1)  direct  typing  by 
providers,  (2)  dictation — transcriptions  are  pro- 
vided on  disk  and  are  uploaded  for  approval  and 
display,  and  (3)  data  capture  from  other  clinical 
areas  (for  example,  laboratory,  electrocardiogram 
[EKG],  discharge  abstracts).  Security  is  through  a 
password  mechanism;  providers  who  serve  in  the 
outpatient  area  are  given  security  privileges  to 
enter  and  edit  data. 

Comments.  The  system's  data  base  architecture 
supports  research  use.  There  have  not  yet  been 
studies  which  specifically  use  the  ambulatory  record 
information  for  outcomes  evaluation,  although 
there  is  outcomes  research  using  other  areas  of  the 
BICS  data  base.  A  system  of  health-maintenance 
standards  and  reminders,  using  the  computer's 
health-maintenance  data,  is  being  developed. 

System:  BIHS 

Beth  Israel  Hospital  System 
Boston,  MA 

Primary  Contact:  C.  Safran,  M.D. 

Since  1976,  the  Beth  Israel  Hospital  System  has 
utilized  a  hospital-wide  clinical  computing  system 
that  supports  inpatient  and  outpatient  services.  It  is 
an  integral  part  of  patient  care  in  the  hospital  and 
was  designed  with  the  primary  goal  of  helping  the 
clinician  care  for  patients.  This  system  for  clinical 
computing  is  one  of  the  most  heavily  used  in  the 
country.  Clinicians  use  the  computer  to  display  test 
results  over  40,000  times  per  week,  send  over  13,000 
pieces  of  electronic  mail,  and  perform  over  1,000  lit- 
erature searches.  In  1989,  the  Outpatient  Medical 
Record  (OMR)  system  was  added  to  the  system, 
and  it  was  specifically  designed  to  encourage  physi- 
cians to  enter  data  directly  into  the  computer. 

Characteristics.  The  hospital  system  runs  on  a  net- 
work of  Data  General  Eclipse  Computers  that  are 
interconnected  and  is  programmed  in  the  MIIS 
dialect  of  MUMPS.  Terminals  interface  with  the 
computer  and  are  located  in  every  physician  office 
and  examination  room.  This  system  was  designed 


to  permit  the  acquisition  of  information  at  the  point 
of  transaction.  The  data  are  then  immediately 
available  to  the  user  who  needs  these  data,  be  it  the 
physician  in  an  ambulatory  office,  the  nurses  at  a 
nursing  station,  or  a  technician  in  the  laboratory. 
The  outpatient  system  supports  appointments, 
vaccine  eligibility  reminders,  flowsheets,  progress 
notes  (indexed  to  problems  on  problem  list),  and 
retrieval.  Programs  allow  clinicians  to  enter  and 
edit  data  and  display  problem  lists,  medications 
lists,  and  health  promotion  and  disease  prevention 
screening  flowsheets.  A  confidentiality  module 
restricts  access  to  the  system. 

A  key  design  feature  of  the  Beth  Israel  Hospi- 
tal's clinical  computing  system  is  that  there  is  a 
common  registry  of  patients;  the  system  is  truly 
integrated.  The  ambulatory  data  base  can  be  shared 
with  other  systems  throughout  the  hospital  and  vice 
versa.  In  practical  terms,  the  results  of  the  PAP 
smear  from  the  cytology  system,  or  the  results  of 
the  mammogram  from  the  radiology  system,  are 
available  to  the  primary  care  screening  programs; 
physicians  do  not  have  to  reenter  these  data.  In  a 
similar  fashion,  physicians  in  the  emergency  room 
can  immediately  view  a  patient's  problems  and  cur- 
rent medications. 

The  OMR  system  currently  has  problem  files 
and  medication  files  on  more  than  7,000  patients. 
Almost  90  percent  of  the  faculty  (15/17)  and  over 
70  percent  of  the  medical  residents  (35/48)  rou- 
tinely enter  problem  lists  and  medication  lists  for 
the  patients  they  see.  Less  than  40  percent  of  the 
interns  (9/24)  routinely  chart  problems  and  medi- 
cations on  the  computer. 

Notes  and  letters  that  are  dictated  by  faculty 
physicians  are  now  transcribed  onto  the  system. 
Additionally,  clinicians  type  over  700  progress  notes 
each  month  directly  into  the  computer. 

Comments.  Studies  show  that  physicians  type  twice 
as  much  on  the  computer-based  problem  lists  than 
they  wrote  in  the  paper  counterpart.  These  studies 
do  not  have  any  data  to  suggest  why  a  minority 
of  clinicians  still  do  not  elect  to  use  the  computer 
system. 

Both  process  and  outcome  variables  for  all 
patients  are  routinely  recorded.  Ambulatory  ser- 
vices and  charges  the  patients  incur  (for  example, 
emergency  room  visits,  number  of  tests,  medication 
dispensed)  are  recorded  as  well  as  hospitalizations, 
discharge  diagnoses,  procedures,  charges,  and  dis- 
charge destination.  The  ambulatory  practice  at  the 
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BIH  is  committed  to  a  complete  transition  to  a 
paperless  record,  and  the  above  data  suggest  this 
process  is  well  underway. 

System:  CIS 

Clinical  Information  System 
Columbia-Presbyterian  Medical  Center  (CPMC) 
New  York,  NY  ' 

Primary  Contact:  P.  Clayton,  Ph.D. 

CPMC  is  developing  a  clinical  information  sys- 
tem (CIS)  with  decision  support  applications  that 
continuously  monitor  patient  data  as  they  are  added 
to  a  patient-oriented  clinical  data  base.  In  addition 
to  a  longitudinal  patient  data  base,  the  system  also 
contains  a  medical  entities  dictionary,  a  collection  of 
Medical  Logic  Modules  (MLMs),  and  a  common 
results  review  function. 

Because  CIS  is  part  of  an  Integrated  Academic 
Information  Management  System  (IAIMS)  being 
developed  at  CPMC,  the  user  can  also  access  uni- 
versity and  hospital  administrative  functions,  basic 
research  applications,  and  library  services  from  the 
same  workstation  which  is  used  to  access  clinical 
data.  There  are  over  50  hosts  or  servers  and  1,000 
workstations  on  the  CPMC/IAIMS  network.  The 
system  is  heavily  used  for  patient  care;  on  a  daily 
basis,  there  are  over  7,000  inquiries  by  over  1,000 
individuals. 

Characteristics.  CIS  keeps  all  data  for  patients  avail- 
able online  indefinitely  regardless  of  whether  the 
patient  is  seen  in  the  hospital,  emergency  room,  or 
outpatient  clinic.  Data  are  acquired  from  other  clin- 
ical systems  (clinical  laboratory,  radiology,  phar- 
macy, and  so  on)  via  Health  Level  Seven  communi- 
cations. Most  of  the  current  results  are  narrative 
rather  than  coded,  but  the  new  applications  that  are 
being  developed  and  added  use  increasing  amounts 
of  coded  data.  There  is  a  minor  amount  of  physician 
data  entry.  The  central  patient  data  base  has  a  small 
number  of  tables,  each  with  a  small  number  of  gen- 
eral purpose  columns.  A  controlled  vocabulary  of 
"codes"  and  values  are  stored  in  the  general  pur- 
pose columns  to  represent  various  clinical  data 
items  such  as  diagnostic  findings,  vital  signs,  and 
medications.  This  philosophy  was  adopted  to  permit 
online  decisionmaking  and  to  accommodate  the 
growth  expected  as  more  of  the  patient  record 
becomes  computer-based.  One  of  the  defining  char- 
acteristics of  the  system  is  a  Medical  Entities  dic- 
tionary which  is  based  on  extensions  of  the  National 


Library  of  Medicine  UMLS.  The  definition  of 
vocabulary  terms  supports  multiple  parents  for  an 
entity,  codes  for  an  entity  that  may  have  many 
synonyms,  and  inheritance  of  properties. 

The  clinical  data  base  (which  started  on  a  hierar- 
chic proprietary  PCS-ADS  platform)  is  now  built  on 
a  relational  platform  with  an  extra  level  of  indirec- 
tion to  optimize  clustering  of  data  by  patient  and  to 
allow  new  entities  to  be  stored  in  the  data  base 
without  modifying  tables.  The  calls  to  this  data  base 
are  based  on  structured  query  language  (SQL). 

Interpatient  queries  (population  analyses)  are 
accomplished  by  periodically  downloading  extracts 
from  the  patient-oriented  data  base  into  relational 
data  bases  with  more  traditional  design. 

Data  are  gathered  from  multiple  ancillary  sys- 
tems but  are  stored  in  a  central  data  base  which 
supports  a  common  results  review  program.  In  the 
instance  of  failure  of  one  of  the  data  base  hosts,  an 
automatic,  redundant  network  path  to  the  primary 
source  assures  that  the  data  are  always  available. 
The  decisionmaking  system  and  data  base  reside 
on  IBM  mainframes  but  have  been  designed  to  be 
portable. 

Comments.  CIS  can  be  used  for  online  decision- 
making and  has  a  growing  capacity  for  medical 
records  and  interpatient  queries.  This  system 
demonstrates  a  potential  for  medical  effectiveness 
research. 

System:  COSTAR 

Computer-Stored  Ambulatory  Record 
Laboratory  of  Computer  Science 
Massachusetts  General  Hospital 
Boston,  MA 

Primary  Contact:  J.  Campbell 

COSTAR  was  developed  by  the  Laboratory  of 
Computer  Science  at  Massachusetts  General  Hospi- 
tal and  was  implemented  in  1968  at  the  Harvard 
Community  Health  Plan  (HCHP).  It  was  one  of  the 
earliest  systems  capable  of  producing  an  automated 
computer-based  patient  record  designed  to  replace 
the  paper  medical  record.  This  modular  medical 
information  and  record  system  is  available  commer- 
cially and  is  in  the  public  domain.  It  is  one  of  the 
most  widely  used  automated  ambulatory  medical 
records  systems. 

The  system  is  distributed  and  enhanced  by  the 
COSTAR  Users  Group  which  has  a  mailing  list  of 
750  and  a  current  membership  of  more  than  100 
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individuals  and  organizations.  The  country  of  Malta 
has  implemented  COSTAR  as  the  national  public 
health  record,  and  a  network  of  welfare  offices  in 
southern  California  uses  COSTAR  as  the  sole 
record  resource.  There  are  between  200  and  500 
sites  in  the  United  States,  Canada,  Europe,  and 
Australia.  An  exact  tally  of  sites  is  impossible 
because  of  the  public  domain  distribution  policy. 

Characteristics.  The  system,  developed  between 
1968  and  1978,  uses  MUMPS  on  both  VAXs  and 
personal  computers  (PCs).  It  is  modular  in  design, 
permitting  system  tailoring  and  phased  implementa- 
tion. The  six  modules  include:  (1)  security  and 
integrity,  (2)  registration,  (3)  scheduling,  (4)  medical 
record,  (5)  billing  and  accounts  receivable,  and  (6) 
Medical  Query  Language  (MQL).  MQL  is  a  useful 
resource  for  COSTAR  but  is  not  part  of  the  public 
domain  package  and  requires  a  site  license  to  install. 
Individual  sites  can  select  the  portions  of  the  system 
they  wish  to  install.  Current  release  5.9  has  under- 
gone periodic  revisions  since  the  first  public  domain 
copy  in  the  1970s  and  now  includes  an  automated 
reminder  system.  The  system  supports  codified  clin- 
ical information  in  seven  major  divisions:  physical 
findings,  diagnoses,  laboratory  tests,  medications, 
nonmedical  therapies,  procedures,  and  personal  his- 
tory. COSTAR  has  a  dictionary  of  medical  terms 
that  each  user  group  may  define  for  its  own  needs. 
Each  element  in  the  dictionary  corresponds  to  a 
unique  COSTAR  code. 

Computer-printed  encounter  forms  are  used  by 
clinicians  for  data  capture.  Administrative  and 
medical  data  are  included  on  the  self-encoded 
forms.  Office  visits,  telephone  calls,  home  visits, 
and  hospital  stays  are  entered  into  the  system 
either  through  the  encounter  form  or  from  narra- 
tive dictation.  The  narratives  and/or  encounter 
forms  are  entered  by  medical  records  department 
clinical  personnel.  MQL  supports  complex  searches 
of  the  data  base  and  retrieval. 

Comments.  There  are  some  concerns  about  the  cost 
of  COSTAR  in  private  practice  and  small  clinic  set- 
tings (Dambro,  Weiss,  McClure,  and  Vuturo,  1988), 
although  it  has  been  argued  that  not  all  benefits 
were  factored  into  the  cost  estimate  (Tierney  and 
McDonald,  1991).  COSTAR  has  served  as  a  base 
for  many  medical  records  developments  including 
public  and  occupational  health  systems,  veterinary 
practice  systems,  and  problem-oriented  records  sys- 
tems. Research  groups  have  developed  within  the 


COSTAR  community.  The  largest  effort  dedicated 
to  medical  research  interests  is  a  collaboration  of  30 
sites  in  the  United  States  and  Canada  conducting 
research  on  multiple  sclerosis.  This  collaborative 
effort  has  resulted  in  a  data-sharing  agreement, 
pooled  research  projects,  and  pending  publications. 
The  system  has  been  used  for  clinical  studies 
(Payne,  Goroll,  Morgan,  and  Barnett,  1990)  and  can 
support  medical  effectiveness  research. 

System:  DIOGENE 

Division  Informatique  Hopital  Geneve 
University  Hospital  of  Geneva 
Geneva,  Switzerland 
Primary  Contact:  J.  Scherrer,  M.D. 

DIOGENE  is  a  government-sponsored  hospital 
information  system  with  goals  to  reduce  average 
length  of  stay  and  improve  operations.  It  was  intro- 
duced in  1974  with  applications  added  through  1988. 
It  is  a  comprehensive  hospital  information  system  of 
automated  medical  records.  DIOGENE  supports 
inpatient  and  outpatient  services  and  utilizes  a  fully 
integrated  data  base. 

Characteristics.  DIGENE  is  housed  in  2  CYBER 
180/845s,  9  satellite  computers  operating  under 
UNIX,  113  ward  unit  stations,  and  454  connected 
personal  computers  and  terminals.  It  is  written  in 
BABEL,  a  locally  developed  language,  and  is  sup- 
ported by  an  Intersat  Network  that  links  all  comput- 
ers in  the  system.  The  system  supports  all  hospital 
data  functions.  Security  procedures  involve  a  strict 
protocol  for  access  to  data. 

The  system  interface  is  an  operator  pool  that 
enters  data  or  processes  requests  for  information. 
Nurses  and  other  clinicians  call  the  operator  from 
the  ward  or  clinic  via  a  telestation  and  update  all 
information.  The  operator  enters  the  data  directly, 
and  the  entered  information  is  displayed  on  the  ter- 
minal. At  the  completion  of  the  dictation,  the  clini- 
cian suggests  any  changes  and  then  approves  the 
text  which  becomes  a  permanent  part  of  the  com- 
puter-based patient  record.  Information  is  retrieved 
in  a  similar  fashion  and  displayed  on  the  terminal. 
Recently,  in  a  pilot  study,  physicians  entered  diag- 
noses and  procedures  directly  into  the  system. 

Comments.  Since  this  system  contains  an  automated 
medical  record,  it  would  seem  most  useful  for  clini- 
cal and  medical  effectiveness  studies.  Documenta- 
tion reviewed  thus  far,  however,  does  not  address 
research  uses  of  the  system. 
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System:  HCHP/AMRS 

Harvard  Community  Health  Plan  Ambulatory 

Medical  Record  System,  Brookline,  MA 
Primary  Contact:  K.  Coltin 

AMRS  was  adapted  from  the  Computer-Stored 
Ambulatory  Record  (COSTAR)  system  developed 
by  the  Laboratory  of  Computer  Science  at  Massa- 
chusetts General  Hospital.  After  COSTAR  was 
used  for  10  years,  a  major  change  in  the  technology 
of  the  HCHP  system  was  implemented  in  1978; 
however,  the  functionality  of  the  COSTAR  system 
(Version  IV)  was  essentially  replicated  in  the  new 
AMRS  system. 

AMRS  currently  functions  as  the  ambulatory 
medical  record  for  patient  care  encounters  occur- 
ring in  11  health  centers  serving  approximately 
250,000  active  members  and  providing  approxi- 
mately 1.5  million  office  visits  per  year.  Historical 
medical  information  is  available  for  over  800,000 
patients  for  varying  time  intervals  over  20  years. 
Hospitalization  data  are  not  captured  in  AMRS  on 
a  regular  basis  and  are  routinely  available  only  from 
a  separate  automated  claims  system. 

Characteristics.  The  AMRS  system  was  housed  on 
12  DEC  PDP  ll/84s  networked  through  Ethernet 
to  each  other  and  to  a  VAX  server.  The  AMRS  sys- 
tem was  to  be  connected  to  VAX  hardware  in  the 
spring  of  1992.  The  operating  system  on  the  PDPs 
was  ISM  M/l  1+  (Version  3.4),  and  the  language  was 
Standard  MUMPS. 

Through  the  VAX  server,  the  AMRS  network  is 
also  networked  to  separate  enrollment/billing,  labo- 
ratory information,  and  referral/authorization  sys- 
tems. All  of  these  systems  feed  data  electronically 
into  the  AMRS  patient  record. 

Data  on  diagnoses/problems,  therapies,  and  pro- 
cedures occurring  at  health  center  encounters  are 
recorded  by  the  clinician  on  paper  encounter  forms, 
which  are  encoded  and  input  by  medical  records 
personnel.  In  addition  to  the  coded  problems,  thera- 
pies, and  procedures,  both  free  text  comments  (up 
to  180  characters)  and  longer,  dictated  notes  are 
code-linked  and  input  to  AMRS. 

Medications  prescribed  at  health  center  visits  or 
by  telephone  are  recorded  by  the  clinician  in  the 
therapies  section  of  the  encounter  form  and  input  by 
medical  records  personnel  to  AMRS,  and  prescrip- 
tions filled  at  HCHP  pharmacies  are  input  by  phar- 
macists to  a  separate  automated  pharmacy  system. 


Laboratory  and  x-ray  orders  are  recorded  by 
clinicians  on  a  laboratory  or  radiology  request  form. 
Radiology  results  are  input  directly  to  AMRS,  and 
laboratory  requests  and  results  are  input  to  a  labo- 
ratory information  system  and  fed  electronically 
into  AMRS.  Results  for  outpatient  tests  performed 
outside  the  health  center  are  sent  to  both  the  order- 
ing clinician  and  the  medical  records  department, 
where  they  are  entered  into  AMRS. 

Query  and  data  retrieval  functions  on  AMRS 
are  supported  by  the  Clinical  Analysis  Support  Sys- 
tem (CLASS)  which  has  been  developed  and 
enhanced  by  HCHP  over  the  past  12  years.  The 
CLASS  query  software  permits  the  identification  of 
an  enrollee/patient  population  on  the  basis  of 
defined  demographic,  enrollment,  utilization,  and/or 
clinical  code  characteristics.  Various  data  output 
options  (for  example,  lists,  labels,  full  medical 
records  or  selected  excerpts,  reports,  files)  may  then 
be  produced  for  the  defined  population.  CLASS 
includes  a  file  extract  utility  which  produces  pre- 
defined, fixed  format  text  files.  Either  ASCII  text 
files  for  downloading  to  a  PC  or  ASCII  or  EBCDIC 
text  files  written  to  magnetic  tape  may  be  produced. 
Several  different  file  extracts  are  available  (for 
example,  medical  encounter  data,  prenatal  data, 
membership/demographic  data,  laboratory  test 
result  data).  These  files  can  be  read  by  almost  any 
PC  or  mainframe  software  package. 

Similar  query  and  file  extract  utilities  have  been 
developed  on  the  automated  claims  system  for 
downloading  hospital  and  outside  outpatient  claims 
data  to  PCs. 

Routine  file  extracts  from  the  HCHP  automated 
pharmacy  system  are  written  to  magnetic  tape  for 
analysis  on  a  mainframe  system  at  Brigham  and 
Women's  Hospital,  as  part  of  a  Joint  Pharmaco- 
epidemiology Research  Program. 

Comments.  Numerous  research  and  management 
studies  have  been  conducted  utilizing  HCHP  data 
files  which  were  extracted  from  these  diverse  and 
technically  incompatible  clinical  information  sys- 
tems and  subsequently  linked  and  merged  for  analy- 
sis in  either  a  PC  or  mainframe  environment.  DBase 
IV  and  PC-SAS  have  been  used  extensively  with 
these  files  on  PCs,  and  SAS  and  SPSS  have  been 
used  on  mainframes. 
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System:  HELP 

Health  Evaluation  Through  Logical  Processing 
University  of  Utah  and  Intermountain 

Health  Care  Corporations 
LDS  Hospital 
Salt  Lake  City,  UT 
Primary  Contact:  T.A.  Pryor,  M.D. 

HELP  an  integrated  hospital  information  sys- 
tem, designed  to  serve  both  the  clinical  and  adminis- 
trative needs  of  the  hospital,  has  been  in  develop- 
ment and  use  since  the  late  1950s  and  early  1960s. 
The  primary  objective  of  HELP  is  to  provide  med- 
ical decision  support.  It  has  a  flexible  medical  dictio- 
nary and  decision  support  imbedded  in  the  system. 

Characteristics.  The  system  is  a  common  integrated 
data  base  system  with  terminals  in  every  nursing 
division.  In  1981,  it  was  converted  to  Tandem  com- 
puters. The  data  base  is  hierarchical  in  structure. 
There  are  plans  to  move  toward  distributed  process- 
ing and  a  micro-based  HELP  system  in  physicians' 
offices  for  followup.  The  system  contains  more  than 
100,000  rules  and  processes  pertaining  to  a  broad 
spectrum  of  health  care  to  assist  in  decisionmaking. 
The  system  contains  a  language  for  processing  and 
assisting  with  medical  decisions.  It  supports  admis- 
sions, discharges,  transfer,  order  entry,  medical 
records,  and  clinical  data. 

Professionals,  primarily  nurses,  enter  quantita- 
tive and  procedural  data  directly  into  the  system 
using  terminals.  In  addition,  HELP  interfaces  with 
financial,  laboratory,  and  intensive  care  unit  moni- 
toring systems.  Only  minimal  history  and  physical 
examinations  are  contained  in  the  HELP  record. 
Outpatient  data  are  captured,  although  these  data 
may  be  limited  reflecting  the  limited  outpatient  set- 
tings of  the  hospital.  Retrieval  is  supported  through 
a  query  system  called  STRATO. 

Comments.  HELP,  although  primarily  a  hospital 
information  system,  can  support  medical  effective- 
ness research,  since  the  hospital's  outpatient  clinics 
are  included  in  the  system.  As  care  evolves  to  an 
outpatient  setting,  physician-office  interfaces  are 
planned  for  the  system.  The  system's  emphasis  on 
decisionmaking  support  should  be  of  interest  to 
medical  effectiveness  researchers. 


System:  MARS 

Medical  Archival  System 
University  of  Pittsburgh 
Pittsburgh,  PA 

Primary  Contact:  J.K.  Vries,  M.D. 

MARS  is  a  large-scale  medical  data  archiving 
system  that  integrates  patient  data  from  central 
transcription,  laboratories,  pharmacy,  radiology,  and 
other  departmental  systems.  It  was  implemented  in 
1989  in  both  inpatient  and  outpatient  environments, 
and  its  development  continues. 

Characteristics.  MARS  is  a  distributed,  parallel 
processing  data  archival  and  retrieval  system.  It  has 
a  fiberoptic  data  communications  network  running 
on  a  heterogeneous  network  of  45  computers  serving 
over  800  terminals  (2  DEC  System  3100  and  4  micro 
VAX  II  computers)  and  32  Unix  workstations. 

Data  are  entered  from  transcription  and  dictated 
report  files,  as  well  as  departmental  systems.  Plans 
are  to  enter  nursing  protocols  also. 

Medical  records  are  online.  Data  base  users  enter 
searches  or  other  commands  at  user  interface  work- 
stations and  computers  that  execute  the  MARS 
client  interface  processes  which  in  turn  search  the 
data  base  and  provide  the  outputs  (client-server 
model).  Although  a  full  range  of  retrieval  options  is 
not  yet  available,  data  can  be  extracted  for  links  to 
statistical  packages  and  research. 

Comments.  This  system  is  still  under  development 
using  state-of-the-art  technology.  Plans  are  to  con- 
tinue development  of  programs  for  automated  med- 
ical records  coding  and  for  generating  International 
Classification  of  Diseases,  9th  Revision,  Clinical 
Modification  (ICD-9-CM)  codes  or  QMR  (Quick 
Medical  Reference)  diagnoses,  and  procedures  cod- 
ing. Theoretically,  this  system  can  support  medical 
effectiveness  research. 

System:  PIMS/PIPS 

Patient  Information  Management  System/ 

Patient  Information  Protocol  System 
Loyola  University  Medical  Center 
Maywood,  IL 

Primary  Contact:  A.  Chandrasekhar,  M.D. 

Development  of  the  Patient  Information  Man- 
agement System  (PIMS)  and  Patient  Information 
Protocol  System  (PIPS)  started  in  1989  within  the 
Department  of  Medicine  at  Loyola  University's 
Stritch  School  of  Medicine.  The  system's  design 
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goals  were  to  integrate  departmental  and  host- 
based  patient  data  into  an  electronic  medical 
record  for  outpatient  care  and  longitudinal  clinical 
research  purposes.  As  of  1991,  the  system  con- 
tained information  on  a  patient  population  of 
approximately  80,000  and  had  67  defined  disease 
or  research  protocols. 

Characteristics.  The  system  is  based  on  Revelation 
Technologies'  data  base  management  system — 
Advanced  Revelation  (AREV).  The  system  oper- 
ates within  a  medical  center-wide  local  area  net- 
work (LAN)  environment  with  approximately 
three  gigabytes  of  data  distributed  among  four 
Intel  386/486  file  servers.  The  entire  LAN  operates 
under  Novell  Netware  2.15c  and  3.11.  The  network 
supports  approximately  250  workstations  with  70 
simultaneous  data  base  users.  Specific  interfaces 
for  PIMS/PIPS  have  been  developed  to  support 
the  electronic  capture  of  clinical  laboratory  and 
admission/discharge  information.  The  LAN  envi- 
ronment and  data  base  management  system  pro- 
vide multiple  layers  of  security  and  audit  logging. 
Users  can  be  restricted  to  specific  accounts,  physi- 
cal workstations,  and  time  periods.  The  system 
supports  online  data  collection  and  codification 
through  a  wide  range  of  user-defined  tables  (for 
example,  Current  Procedural  Terminology  [CPT] 
and  ICD-9).  The  system  also  supports  ad  hoc 
queries  and  reports  through  AREV's  English-like 
command  language  and  SQL. 

Physicians  and  support  staff  enter  data  directly 
into  the  system  using  predefined  menus  and  collec- 
tion windows.  Integrated  modules  include  demo- 
graphics, current  medications,  problem  list,  prior 
events,  clinical  laboratory  data,  reminders,  and 
online  procedures.  Prior  events  and  problem  lists 
can  be  captured  automatically  through  interfaces 
with  admissions,  procedure,  and  charge  capture  sys- 
tems. Users  can  initiate  electronic  capture  of  clinical 
laboratory  data  or  may  schedule  automatic  daily 
captures  at  predefined  times.  Patient  data  can  be 
reported  in  a  wide  range  of  summary  formats. 

PIPS  allows  clinicians  to  define  information  pro- 
tocols that  address  patient  care  practices  or  specific 
research  projects.  Protocols  are  a  collection  of  data 
objects  from  user-defined  categories  (for  example, 
physical  findings,  drugs  or  treatments,  symptoms, 
toxicities,  laboratory  data,  x-ray/scans,  and  lesions). 
Protocols  allow  for  time-oriented  collection  and 
review  of  patient  data  and  support  data  entry  as  indi- 
vidual values  or  full  text  progress  notes.  Protocol 


data  summaries  can  be  extracted  for  analysis  using 
external  statistical  packages  (for  example,  SAS). 

Comments.  Since  one  of  the  goals  of  this  system  is 
to  support  longitudinal  studies,  it  would  appear  that 
it  could  be  used  for  medical  effectiveness  research. 

System:  RMRS 

Regenstrief  Medical  Records  System 
Indiana  University  Medical  Center 
Indianapolis,  IN 

Primary  Contact:  Clement  McDonald,  M.D. 

This  system  began  in  1972  with  the  goal  of  creat- 
ing a  "summary"  computer-stored  medical  record. 
RMRS  now  contains  records  for  all  420,000  regis- 
tered patients  of  Wishard  Memorial  Hospital's  out- 
patient clinics  as  well  as  patients  of  a  neighboring 
Veterans  Affairs  hospital  and  Indiana  University 
Hospital.  The  system  processes  laboratory  tests, 
inpatient  drug  orders,  outpatient  prescriptions,  and 
scheduling  and  contains  the  full  coded  diagnostic 
impression  for  most  diagnostic  studies.  The  full  test 
is  also  available  for  some  (for  example,  diagnostic 
radiology).  Data  are  fully  coded  and  defined  in  a 
data  dictionary.  Medical  records  are  time-oriented. 
The  system  integrates  problem  lists,  inpatient  and 
outpatient  records,  charges,  immunizations,  and 
duration  of  visits. 

Characteristics.  The  system  is  housed  in  cluster  DEC 
VAXs  to  8550  and  386  PC  workstations,  integrated 
through  Ethernet.  Languages  include  VAX  BASIC 
and  VAX  Assembly  Language.  Microcomputers  use 
Revelation  R/Basic  and  Intel  Assembly  language. 
The  medical  records  data  base  resides  in  Digital 
Equipment's  Record  Management  System. 

RMRS  is  part  of  a  larger  administrative  support 
system  that  handles  appointment  scheduling  and 
charge  capture.  Medical  records,  prescriptions,  nar- 
ratives, visits,  laboratory  records,  and  retrieval  are 
supported  through  the  system. 

Nurse-  and  physician-collected  data  are  captured 
on  computer  generated,  time-oriented  encounter 
forms  and  special  multiple  choice  forms,  and  these 
data  are  entered  by  data  entry  specialists;  however, 
physicians  in  the  General  Medicine  Clinic  and  the 
inpatient  internal  medicine  service  enter  orders 
directly  through  workstations.  There  are  pharmacy 
and  laboratory  data  and  clinic  computer  subsystems 
within  the  RMRS.  Tapes  from  the  hospital  case 
abstracting  service  also  are  entered  in  an  automated 
manner.  Radiologic  procedures,  EKGs,  occult 
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blood,  PPD  (purified  protein  derivative)  tests, 
spirometries,  bone  marrow  diagnoses,  cardiac 
echoes,  vital  signs,  and  weights  are  coded  and 
entered  manually  by  specially  trained  staff.  The 
nature  and  location  of  a  lesion  are  entered  as  well  as 
trend  data  ("better"  or  "worse"),  but  not  the  details 
of  the  size  and  shape  of  each  finding.  Full  texts  of 
x-ray  or  surgical  pathology  reports  and  hospital 
admission  histories  and  discharge  summaries  are 
also  stored  in  the  computer  in  a  separate  file. 

Queries  and  data  retrieval  are  handled  in  two 
ways:  through  a  fast  retrieval  module  within  RMRS 
and  through  CARE,  a  medical  query  language.  The 
fast  retrieval  module  does  not  give  an  indication  of 
time  sequence  and  is,  therefore,  not  appropriate 
for  research.  CARE  serially  searches  the  medical 
records  and  can  be  used  for  quality  assurance  as 
well  as  research.  An  extract  utility  interfaces  with 
PC  and  mainframe  SAS  and  BMDP. 

Comments.  This  system  was  designed  to  automate 
the  medical  record  and  has  become  a  hospital 
administrative  and  clinical  information  system. 

Over  70  million  observation  records  that  detail 
clinical  observations  over  time  for  more  than 
500,000  patients  from  3  hospitals  and  a  network  of 
inner  city  clinics  are  stored.  The  data  have  been 
used  to  support  research,  and  the  system  can  be 
used  to  support  studies  of  medical  effectiveness.  It 
is  particularly  noted  for  providing  computer-gener- 
ated reminders  to  physicians  about  clinical  events 
that  may  require  corrective  action. 

System:  RPMS 

Resource  and  Patient  Management  System 
Indian  Health  Service  (IHS) 
Office  of  Health  Program  Research 

and  Development 
ADP  Systems  Support  Division 
Tucson,  AZ 

Primary  Contact:  Bill  Mason 

An  integrated  group  of  automated  data  systems 
designed  to  operate  on  micro-  and  minicomputers 
located  in  any  IHS  or  tribal  health  facility  (that  is, 
hospital  or  full-time  clinic),  the  RPMS  of  the  IHS 
will  link  health  facilities  to  each  other  and  to  admin- 
istrative units  such  as  area  offices.  The  main  objec- 
tive is  to  integrate  patient  care  and  cost  data  in  a  sin- 
gle automated  data  processing  system  that  collects 
and  stores  a  core  set  of  health  and  management  data 
which  cuts  across  disciplines  and  facilities. 


Characteristics.  A  typical  RPMS  configuration  in  a 
health  facility  might  include  the  following  systems: 
patient  registration,  pharmacy,  dental,  maternal 
and  child  health,  contract  health  services,  labora- 
tory, and  the  patient  care  component  (PCC).  The 
PCC  provides  for  the  collection  and  storage,  on 
local  RPMS  computers,  of  a  broad  range  of  health 
data  resulting  from  inpatient,  outpatient,  and  field 
visits  at  IHS,  tribal,  contract,  and  community  sites. 
It  is  designed  to  support  health  care  delivery,  plan- 
ning, management,  and  research.  For  health  profes- 
sionals, the  PCC  is  a  tool  that  assists  in  providing 
the  type  of  care  that  addresses  all  of  a  patient's 
known  health  problems  and  preventive  health 
needs.  Planners  use  information  such  as  the  num- 
bers of  and  reasons  for  various  types  of  visits.  Man- 
agers use  aggregate  information  such  as  numbers  of 
patients  who  have  insurance  and  the  types  of  insur- 
ance. Researchers  use  the  information  for  a  variety 
of  locally  designed  projects.  PCC  data  types 
include:  date,  type  and  location  of  visit,  providers 
of  service,  measurements,  diagnoses  and  proce- 
dures, health  problems  and  treatment  plans,  per- 
sonal and  family  history,  reproductive  factors,  labo- 
ratory test  results,  and  a  variety  of  other 
health-related  information.  As  with  other  RPMS 
components,  the  PCC  utilizes  the  File  Manager 
data  base  management  system  for  data  storage. 

Key  features  of  the  PCC  include:  integrated  data 
files,  encounter  forms,  data  entry  module,  a  com- 
prehensive health  summary,  patient  registers,  report 
generation,  data  transmission  to  the  IHS  data  cen- 
ter, online  help  screens,  and  a  diagnosis-related 
grouper  program. 

Comments.  Although  RPMS  is  one  of  the  few  auto- 
mated ambulatory  medical  record  systems  available, 
it  is  not  well  known  because  support  for  the  system 
has  been  insufficient  to  allow  widespread  dissemina- 
tion of  information  about  RPMS. 

System:  STOR 

Summary  Time-Oriented  Record 
University  of  California  at  San  Francisco 
San  Francisco,  CA 

Primary  Contact:  Q.E.  Whiting-O'Keefe,  M.D. 

STOR  is  a  clinical  information  system  that  has 
operated  in  an  inpatient  and  outpatient  environ- 
ment at  UCSF  since  1978.  In  1985,  it  was  approved 
for  clinic-wide  implementation  in  the  university's 
ambulatory  care  clinics.  In  1988,  STOR  contained 
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60,000  outpatient  medical  records  and  supported  22 
clinic  locations  with  200,000  visits  per  year. 

Characteristics.  The  system  has  been  translated 
from  MIIS  into  the  MUMPS  language  (1989)  and  is 
housed  in  a  Data  General  MV  40000.  It  uses  LAN 
technology  and  a  hierarchical  record  structure.  It 
supports  online  real-time  display  for  accessing  clini- 
cal information.  STOR  gathers  its  data  from  eight 
existing  ancillary  computer  systems. 

STOR  is  a  patient  care-oriented  system  that 
replaces  the  paper  ambulatory  medical  record  in  85 
percent  of  the  patient  visits  and  augments  it  the  rest 
of  the  time.  A  master  problem  and  therapy  list  was 
added  in  1990.  It  supports  a  generalized  problem 
and  manifestation  list,  the  elements  of  which  are 
related  in  a  hierarchical  structure. 

STOR  contains  a  centralized  summary  data  base 
of  clinical  information  gathered  from  ancillary  or 
departmental  computer  systems.  Patient  data  are 
collected  by  providers  using  one  of  several  com- 
puter-generated data  display  and  capture  encounter 
instruments  that  specialize  according  to  clinical 
view,  diagnosis,  and  clinician.  These  data  are  tran- 
scribed and  entered  manually  into  the  system. 

STOR's  interpatient  retrieval  system,  called 
SEARCH,  has  recently  been  upgraded  to  a  com- 
plete query  language. 

Comments.  This  system  has  supported  research  and 
has  been  evaluated  with  positive  results  regarding 
accuracy  of  the  medical  record.  It  would  appear  that 
it  can  be  used  to  support  medical  effectiveness 
research. 

System:  THERESA 

Grady  Memorial  Hospital 
Atlanta,  GA 

Primary  Contact:  D.  Brizenbine 

Development  of  THERESA  was  initiated  in 
1981  at  the  Emory  University  School  of  Medicine's 
primary  clinical  teaching  facility.  As  of  1990, 18  mil- 
lion documents  were  online.  It  evolved  from  a  deci- 
sion-aiding system  called  Medical  Aggregate- 
Record  Inquiry  (MARI).  THERESA  supports 
problem-oriented,  task-oriented,  event-oriented, 
and  time-oriented  views  of  the  patient  record  along 
with  English-like  queries.  It  can  describe  usage  pat- 
terns for  each  individual  drug  item  and  each  cate- 
gory of  drugs  in  the  pharmacy.  In  addition,  the 
system  can  track  numbers  of  tests  performed  along 
with  the  reasons  for  analysis. 


Characteristics.  The  system  is  based  on  a  custom- 
made  object-oriented  data  base  management  sys- 
tem that  was  designed  to  address  problems  associ- 
ated with  the  paper  patient  record.  It  is  housed  in  a 
distributed  network  of  super  minicomputers.  It  was 
developed  by  Medical  Systems  Development 
Corporation  of  Atlanta  and  is  housed  on  four 
VAX  8550s  and  one  VAX  8530  that  support  1,000 
terminals.  Security  is  provided  by  allowing  access 
to  appropriate  users  at  specific  times  and  specific 
hospital  locations.  Further,  the  system  checks  the 
person  who  is  accessing  data  and  can  shut  off  the 
computer  for  unauthorized  access.  The  design  of 
the  system  is  based  on  a  data  dictionary  of  table- 
driven  code  systems  that  supports  an  array  of  file 
and  data  structures. 

Physicians  and  other  clinicians  enter  data 
directly  into  the  system  using  "windowing"  and 
mouse  functions.  Patient  records  include  medical 
history,  physical  examination,  diagnosis,  radiology, 
pharmacy,  orders,  results,  and  progress.  They  also 
contain  physician  and  nursing  notes. 

Comments.  Although  this  system  is  not  described  in 
the  literature  with  the  same  intensity  as  the  other 
academic  systems,  it  is  a  robust  system  and  would 
appear  to  be  capable  of  supporting  medical  effec- 
tiveness studies. 

System:  TMR 

The  Medical  Record 

Duke  University  Medical  Center 

Durham,  NC 

Primary  Contacts:  W.E.  Hammond,  M.D.  and 
D.  Pryor,  M.D. 

TMR  is  a  comprehensive  medical  information 
system  that  supports  patient  care,  administrative 
management,  and  clinical  research.  Its  original 
design  goal  was  to  eliminate  the  paper  medical 
record  in  an  outpatient  setting,  and  it  has  been 
developed  in  an  evolutionary  fashion  over  the  past 
21  years  to  support  inpatient  settings.  It  manages  all 
aspects  of  the  patient  encounter  including  appoint- 
ments/scheduling, diagnostic  workup,  and  treatment 
in  both  the  outpatient  and  inpatient  environments. 
The  Duke  Cardiovascular  Data  Base,  which  has 
supported  numerous  clinical  studies,  was  migrated 
to  TMR  in  recent  years.  TMR  is  used  at  over  25 
sites  in  the  United  States  and  Canada. 
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Characteristics.  TMR  operates  on  DEC  s  VAX 
under  VMS  and  IBM  PCs  with  DOS.  It  uses  the 
GEMISCH  data  base  management  language  and 
has  a  user-controlled  data  dictionary  and  text  table. 
The  system  can  store  and  retrieve  any  data  con- 
tained in  a  traditional  paper  record  with  the  excep- 
tion of  images  such  as  x-rays.  TMR  can  support  a 
complete  list  of  diagnoses  and  procedures,  as  well 
as  subjective  and  physical  findings,  laboratory  data, 
and  therapeutic  interventions. 

Data  can  be  entered  interactively  by  secretaries, 
nurses,  technicians,  and  physicians.  Demographic 
data,  research  protocols,  problems  and  diagnoses, 
subjective  and  physical  findings,  orders,  therapies, 
encounter  data,  and  accounting  data  are  entered. 
Patient  monitoring,  laboratory,  and  other  ancillary 
system  data  are  entered  directly. 

The  systems  retrieval  capabilities  facilitate  mul- 
tiple views  of  the  data  from  a  problem,  time,  task, 
or  encounter  perspective.  The  system  can  generate 
discharge  notes,  referral  letters,  and  other  text 
reports  and  can  initiate  a  protocol  through  decision 
rules.  Queries  are  supported. 

Comments.  TMR  is  used  to  support  research  and 
can  support  medical  effectiveness  studies. 

System:  VIPOR 

Vermont  Integrated  Problem-Oriented  Record 
University  Health  Center  (UHC) 
Burlington,  VT 

Primary  Contact:  T.  Peterson,  M.D. 

VIPOR  was  originally  implemented  in  1986  at 
the  UHC,  the  academic  practice  plan  for  the  Uni- 
versity of  Vermont,  to  assist  in  the  goals  of  patient 
care,  education,  and  research.  UHC  was  the  first 
test  site  for  IDX  Corporation  which  developed  the 
electronic  medical  record.  The  UHC  Version, 
VIPOR,  has  allowed  primary  care  groups  distant 
from  the  central  site  to  share  a  common  patient  data 
base  and  maintain  access  to  laboratory  and  x-ray 
data  electronically.  It  allows  entry  for  both  diction- 
ary (coded)  information  and  free  text  in  a  problem- 
oriented  fashion  facilitating  both  academic  and 
patient  care  applications. 

Eleven  specialty  groups  use  VIPOR,  two  for 
data  entry  and  retrieval,  the  remainder  for  data 
retrieval.  Currently,  there  are  over  200,000  records 
in  the  VIPOR  data  base,  and  UHC  has  383,000 
annual  ambulatory  visits. 


Characteristics.  The  system  runs  on  a  VAX 
8550/VAX  6510  combination  and  is  linked  locally  to 
six  distant  sites  using  an  EtherNet  arrangement;  440 
ports  are  maintained  for  1,200  users.  VIPOR  is  writ- 
ten in  MUMPS  and  can  be  queried  using  MQL. 
Data  entry,  retrieval,  and  report  functions,  as  well  a 
commercial  drug  interaction  program,  are  supported 
by  the  system. 

Data  entry  is  allowed  by  user,  tape,  and  tran- 
scription. Retrieval  is  at  deskside  and  by  designed 
reports.  Sections  include  problems,  notes,  flow 
sheets,  allergies,  medications,  immunizations,  labo- 
ratories (by  tape),  radiology,  and  patient  education. 
Demographic  and  financial  information  are  shared 
with  business  software.  Data  may  be  displayed  in 
secondary  formats  to  aid  interpretation. 

Comments.  The  primary  use  of  this  data  base  thus 
far  has  been  for  clinical  and  educational  purposes. 
It  is  intensively  used  as  the  primary  medical  record 
for  the  Department  of  Family  Practice  and  is  used 
by  the  same  group  to  document  educational  experi- 
ences and  to  facilitate  chart  review  and  audit. 
Family  Practice  and  Radiology  have  used  VIPOR 
for  quality  assurance  studies,  but  no  cost  effective- 
ness or  outcome  analyses  have  been  performed. 
Nonetheless,  the  data  base  is  considered  to  be  a 
unique  and  largely  untapped  resource. 

Solutions  that  will  allow  greater  research  effec- 
tiveness include  expanded  recovery  systems  for  lab- 
oratory data  and  the  use  of  more  sophisticated  data 
search  capabilities  such  as  SQL.  A  statistical  system 
for  online  analyses  or  offline  for  downloaded  files  is 
not  available  at  this  time. 
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